Working with service run on 7860

.gitignore

@@ -0,0 +1,69 @@
+# --- General ---
+.DS_Store
+Thumbs.db
+*.log
+
+# --- Python ---
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+.Python
+.ruff_cache/
+.mypy_cache/
+.pytest_cache/
+.ipynb_checkpoints/
+.cache/
+*.egg-info/
+*.egg
+build/
+dist/
+develop-eggs/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+wheels/
+pip-wheel-metadata/
+.coverage
+.coverage.*
+coverage.xml
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+.python-version
+
+# --- Node / JS ---
+node_modules/
+.pnpm-store/
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+**/dist/
+**/build/
+
+# Keep lockfiles tracked
+!**/package-lock.json
+!**/yarn.lock
+
+# --- IDE ---
+.idea/
+.vscode/
+
+# --- Environment files ---
+.env
+.env.*
+!**/env.example
+!**/.env.example
+
+# --- Example runtime artifacts ---
+examples/voice_agent_webrtc_langgraph/audio_dumps/
+examples/voice_agent_webrtc_langgraph/ui/dist/
+

CHANGELOG.md

@@ -0,0 +1,19 @@
+# NVIDIA Pipecat 0.1.0 (23 April 2025)
+The NVIDIA Pipecat library augments the Pipecat framework by adding additional frame processors and services, as well as new multimodal frames to enhance avatar interactions. This is the first release of the NVIDIA Pipecat library.
+
+## New Features
+
+- Added Pipecat services for [Riva ASR (Automatic Speech Recognition)](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/asr/asr-overview.html#), [Riva TTS (Text to Speech)](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-overview.html), and [Riva NMT (Neural Machine Translation)](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/translation/translation-overview.html) models.
+- Added Pipecat frames, processors, and services to support multimodal avatar interactions and use cases. This includes `Audio2Face3DService`, `AnimationGraphService`, `FacialGestureProviderProcessor`, and `PostureProviderProcessor`.
+- Added `ACETransport`, which is specifically designed to support integration with existing [ACE microservices](https://docs.nvidia.com/ace/overview/latest/index.html). This includes a FastAPI-based HTTP and WebSocket server implementation compatible with ACE.
+- Added `NvidiaLLMService` for [NIM LLM models](https://build.nvidia.com/) and `NvidiaRAGService` for the [NVIDIA RAG Blueprint](https://github.com/NVIDIA-AI-Blueprints/rag/blob/main/docs/quickstart.md).
+- Added `UserTranscriptSynchronization` processor for user speech transcripts and `BotTranscriptSynchronization` processor for synchronizing bot transcripts with bot audio playback.
+- Added custom context aggregators and processors to enable [Speculative Speech Processing](https://docs.nvidia.com/ace/ace-controller-microservice/latest/user-guide.html#speculative-speech-processing) to reduce latency.
+- Added `UserPresence`, `Proactivity`, and `AcknowledgementProcessor` frame processors to improve human-bot interactions.
+- Released source code for the voice assistant example using `nvidia-pipecat`, along with the `pipecat-ai` library service, to showcase NVIDIA services with `ACETransport`.
+
+
+## Improvements
+
+- Added `ElevenLabsTTSServiceWithEndOfSpeech`, an extended version of the ElevenLabs TTS service with end-of-speech events for usage in avatar interactions.
+

CLA.md

@@ -0,0 +1,129 @@
+# Contribution License Agreement
+
+This Contribution License Agreement (“**Agreement**”) is agreed to by the party signing below (“**You**”),
+and conveys certain license rights to NVIDIA Corporation and its affiliates (“**NVIDIA**”) for Your
+contributions to NVIDIA open source projects. This Agreement is effective as of the latest signature
+date below.
+
+## 1. Definitions.
+
+“**Code**” means the computer software code, whether in human-readable or machine-executable form,
+that is delivered by You to NVIDIA under this Agreement.
+
+“**Project**” means any of the projects owned or managed by NVIDIA in which software is offered under
+a license approved by the Open Source Initiative (OSI) (www.opensource.org) and documentation
+offered under an OSI or a Creative Commons license (https://creativecommons.org/licenses).
+
+“**Submit**” is the act of uploading, submitting, transmitting, or distributing code or other content to any
+Project, including but not limited to communication on electronic mailing lists, source code control
+systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
+discussing and improving that Project, but excluding communication that is conspicuously marked or
+otherwise designated in writing by You as “Not a Submission.”
+
+“**Submission**” means the Code and any other copyrightable material Submitted by You, including any
+associated comments and documentation.
+
+## 2. Your Submission. 
+You must agree to the terms of this Agreement before making a Submission to any
+Project. This Agreement covers any and all Submissions that You, now or in the future (except as
+described in Section 4 below), Submit to any Project.
+
+## 3. Originality of Work. 
+You represent that each of Your Submissions is entirely Your original work.
+Should You wish to Submit materials that are not Your original work, You may Submit them separately
+to the Project if You (a) retain all copyright and license information that was in the materials as You
+received them, (b) in the description accompanying Your Submission, include the phrase “Submission
+containing materials of a third party:” followed by the names of the third party and any licenses or other
+restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
+guidelines concerning Submissions.
+
+## 4. Your Employer. 
+References to “employer” in this Agreement include Your employer or anyone else
+for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
+Submission is made in the course of Your work for an employer or Your employer has intellectual
+property rights in Your Submission by contract or applicable law, You must secure permission from Your
+employer to make the Submission before signing this Agreement. In that case, the term “You” in this
+Agreement will refer to You and the employer collectively. If You change employers in the future and
+desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
+and secure permission from the new employer before Submitting those Submissions.
+
+
+## 5. Licenses.
+
+**a. Copyright License**. You grant NVIDIA, and those who receive the Submission directly or
+indirectly from NVIDIA, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
+Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
+the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
+parties.
+
+**b. Patent License**. You grant NVIDIA, and those who receive the Submission directly or
+indirectly from NVIDIA, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
+Your patent claims that are necessarily infringed by the Submission or the combination of the
+Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
+import or otherwise dispose of the Submission alone or with the Project.
+
+**c. Other Rights Reserved**. Each party reserves all rights not expressly granted in this Agreement.
+No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
+granted by implication, exhaustion, estoppel or otherwise.
+
+## 6. Representations and Warranties. 
+You represent that You are legally entitled to grant the above
+licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
+have disclosed under Section 3). You represent that You have secured permission from Your employer to
+make the Submission in cases where Your Submission is made in the course of Your work for Your
+employer or Your employer has intellectual property rights in Your Submission by contract or applicable
+law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
+have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
+You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
+REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
+EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
+PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
+NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
+
+## 7. Notice to NVIDIA. 
+You agree to notify NVIDIA in writing of any facts or circumstances of which
+You later become aware that would make Your representations in this Agreement inaccurate in any
+respect.
+
+## 8. Information about Submissions. 
+You agree that contributions to Projects and information about
+contributions may be maintained indefinitely and disclosed publicly, including Your name and other
+information that You submit with Your Submission.
+
+## 9. Governing Law/Jurisdiction. 
+Claims arising under this Agreement shall be governed by the laws of Delaware, excluding its principles of conflict of laws and the United Nations Convention on Contracts for the Sale of Goods. The state and/or federal courts residing in Santa Clara County, California shall have exclusive jurisdiction over any dispute or claim arising out of this Agreement. You may not export the Software in violation of applicable export laws and regulations.
+
+## 10. Entire Agreement/Assignment. 
+This Agreement is the entire agreement between the parties, and
+supersedes any and all prior agreements, understandings or communications, written or oral, between
+the parties relating to the subject matter hereof. This Agreement may be assigned by NVIDIA.
+
+
+
+
+**Please select one of the options below and sign as indicated.** By signing, You accept and agree to the
+terms of this Contribution License Agreement for Your present and future Submissions to NVIDIA.
+
+___ I have sole ownership of intellectual property rights to my Submissions and I am not making
+Submissions in the course of work for my employer.
+
+- Name (“You”): _________________________________________
+- Signature: _________________________________________
+- Date: _________________________________________
+- GitHub Login: _________________________________________
+- Email: _________________________________________
+- Address: _________________________________________
+
+___ I am making Submissions in the course of work for my employer (or my employer has intellectual
+property rights in my Submissions by contract or applicable law). I have permission from my
+employer to make Submissions and enter into this Agreement on behalf of my employer. By signing
+below, the defined term “You” includes me and my employer.
+
+- Company Name: _________________________________________
+- Signature: _________________________________________
+- By: _________________________________________
+- Title: _________________________________________
+- Date: _________________________________________
+- GitHub Login: _________________________________________
+- Email: _________________________________________
+- Address: _________________________________________

CONTRIBUTING.md

@@ -0,0 +1,14 @@
+# Contributing Guidelines
+
+Use the following guidelines to contribute to this project.
+
+
+## Pull Requests
+Developer workflow for code contributions is as follows:
+
+1. Developers must create a [fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) of this repository for the upstreaming.
+2. Git clone the forked repository and push changes to the personal fork.
+3. Developers must run changes locally to make sure formatting, linting, and unit test checks pass. Check steps for development with the source in [README.md](./README.md) for more details.
+3. Once the code changes are staged on the fork and ready for review, a Pull Request (PR) can be requested to merge the changes from a branch of the fork into a selected branch of upstream.
+4. If you are contributing for the first time, Download [Contribution License Agreement(CLA)](CLA.md) and email a signed CLA to [ttripathi@nvidia.com](mailto:ttripathi@nvidia.com).
+5. Since there is no CI/CD process in place yet, the PR will be accepted and the corresponding issue closed only after adequate testing has been completed, manually, by the developer and/or repository owners reviewing the code.

Dockerfile

@@ -0,0 +1,64 @@
+# Build UI assets
+FROM node:18-alpine AS ui-builder
+
+WORKDIR /ui
+# Install UI dependencies
+COPY examples/voice_agent_webrtc_langgraph/ui/package*.json ./
+RUN npm ci --no-audit --no-fund && npm cache clean --force
+# Build UI
+COPY examples/voice_agent_webrtc_langgraph/ui/ .
+RUN npm run build
+
+# Base image
+FROM python:3.12-slim
+
+# Environment setup
+ENV PYTHONUNBUFFERED=1
+
+# System dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libgl1 \
+    libglx-mesa0 \
+    curl \
+    ffmpeg \
+    git \
+    net-tools \
+    procps \
+    vim \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
+    && pip install --no-cache-dir --upgrade pip uv
+
+# App directory setup
+WORKDIR /app
+
+# App files
+COPY pyproject.toml uv.lock \
+     LICENSE README.md NVIDIA_PIPECAT.md \
+     ./
+COPY src/ ./src/
+COPY examples/voice_agent_webrtc_langgraph/ ./examples/voice_agent_webrtc_langgraph/
+
+# Copy built UI into example directory so FastAPI can serve it
+COPY --from=ui-builder /ui/dist /app/examples/voice_agent_webrtc_langgraph/ui/dist
+
+# Example app directory
+WORKDIR /app/examples/voice_agent_webrtc_langgraph
+
+# Dependencies
+RUN uv sync --frozen
+RUN uv pip install -r agents/requirements.txt
+# Ensure langgraph CLI is available at build time
+RUN uv pip install -U langgraph
+RUN chmod +x start.sh
+
+# Port configuration (single external port for app)
+EXPOSE 7860
+
+# Healthcheck
+HEALTHCHECK --interval=30s --timeout=10s --retries=3 --start-period=60s CMD curl -f http://localhost:7860/get_prompt || exit 1
+
+# Start command
+CMD ["/app/examples/voice_agent_webrtc_langgraph/start.sh"]
+
+

LICENSE

@@ -0,0 +1,24 @@
+BSD 2-Clause License
+
+Copyright (c) 2024–2025, NVIDIA Corporation
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

NVIDIA_PIPECAT.md

@@ -0,0 +1,5 @@
+# NVIDIA Pipecat
+
+The NVIDIA Pipecat library augments [the Pipecat framework](https://github.com/pipecat-ai/pipecat) by adding additional frame processors and services, as well as new multimodal frames to facilitate the creation of human-avatar interactions. This includes the integration of NVIDIA services and NIMs such as [NVIDIA Riva](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/index.html), [NVIDIA Audio2Face](https://build.nvidia.com/nvidia/audio2face-3d), and [NVIDIA Foundational RAG](https://build.nvidia.com/nvidia/build-an-enterprise-rag-pipeline). It also introduces a few processors with a focus on improving the end-user experience for multimodal conversational agents, along with speculative speech processing to reduce latency for faster bot responses. 
+
+The nvidia-pipecat source code can be found in [the GitHub repository](https://github.com/NVIDIA/ace-controller). Follow [the documentation](https://docs.nvidia.com/ace/ace-controller-microservice/latest/index.html) for more details.

README.md

@@ -8,4 +8,85 @@ pinned: false
 short_description: Voice Demos with Ace Controller
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# ACE Controller SDK
+
+The ACE Controller SDK allows you to build your own ACE Controller service to manage multimodal, real-time interactions with voice bots and avatars using NVIDIA ACE. With the SDK, you can create controllers that leverage the Python-based open-source [Pipecat framework](https://github.com/pipecat-ai/pipecat) for creating real-time, voice-enabled, and multimodal conversational AI agents. The SDK contains enhancements to the Pipecat framework, enabling developers to effortlessly customize, debug, and deploy complex pipelines while integrating robust NVIDIA Services into the Pipecat ecosystem.
+
+## Main Features
+
+- **Pipecat Extension:** A Pipecat extension to connect with ACE services and NVIDIA NIMs, facilitating the creation of human-avatar interactions. The NVIDIA Pipecat library augments [the Pipecat framework](https://github.com/pipecat-ai/pipecat) by adding additional frame processors and services, as well as new multimodal frames to enhance avatar interactions. This includes the integration of NVIDIA services and NIMs such as [NVIDIA Riva](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/index.html), [NVIDIA Audio2Face](https://build.nvidia.com/nvidia/audio2face-3d), and [NVIDIA Foundational RAG](https://build.nvidia.com/nvidia/build-an-enterprise-rag-pipeline).
+
+- **HTTP and WebSocket Server Implementation:** The SDK provides a FastAPI-based HTTP and WebSocket server implementation compatible with ACE. It includes functionality for stream and pipeline management by offering new Pipecat pipeline runners and transports. For ease of use and distribution, this functionality is currently included in the `nvidia-pipecat` Python library as well.
+
+## ACE Controller Microservice
+
+The ACE Controller SDK was used to build the [ACE Controller Microservice](https://docs.nvidia.com/ace/ace-controller-microservice/latest/index.html).Check out the [ACE documentation](https://docs.nvidia.com/ace/tokkio/latest/customization/customization-options.html) for more details on how to configure the ACE Controller MS with your custom pipelines.
+
+
+## Getting Started
+
+The NVIDIA Pipecat package is released as a wheel on PyPI. Create a Python virtual environment and use the pip command to install the nvidia-pipecat package.
+
+```bash
+pip install nvidia-pipecat
+```
+
+You can start building pipecat pipelines utilizing services from the NVIDIA Pipecat package. For more details, follow [the ACE Controller](https://docs.nvidia.com/ace/ace-controller-microservice/latest/index.html) and [the Pipecat Framework](https://docs.pipecat.ai/getting-started/overview) documentation.
+
+## Hacking on the framework itself
+
+If you wish to work directly with the source code or modify services from the nvidia-pipecat package, you can utilize either the UV or Nix development setup as outlined below.
+
+### Using UV
+
+
+To get started, first install the [UV package manager](https://docs.astral.sh/uv/#highlights). 
+
+Then, create a virtual environment with all the required dependencies by running the following commands:
+```bash
+uv venv
+uv sync
+source .venv/bin/activate
+```
+
+Once the environment is set up, you can begin building pipelines or modifying the services in the source code.
+
+If you wish to contribute your changes to the repository, please ensure you run the unit tests, linter, and formatting tool.
+
+To run unit tests, use:
+```
+uv run pytest
+```
+
+To format the code, use:
+```bash
+ruff format
+```
+
+To run the linter, use:
+```
+ruff check
+```
+
+
+### Using Nix
+
+To set up your development environment using [the Nix](https://nixos.org/download/#nix-install-linux), follow these steps:
+
+Initialize the development environment: Simply run the following command:
+```bash
+nix develop
+```
+
+This setup provides you with a fully configured environment, allowing you to focus on development without worrying about dependency management.
+
+To ensure that all checks such as the formatting and linter for the repository are passing, use the following command:
+
+```bash
+nix flake check
+```
+
+## CONTRIBUTING
+
+We invite contributions! Open a GitHub issue or pull request! See contributing guildelines [here](./CONTRIBUTING.md).
+

SECURITY.md

@@ -0,0 +1,24 @@
+## Security
+
+NVIDIA is dedicated to the security and trust of our software products and services, including all source code repositories managed through our organization.
+
+If you need to report a security issue, please use the appropriate contact points outlined below. **Please do not report security vulnerabilities through GitHub.**
+
+## Reporting Potential Security Vulnerability in an NVIDIA Product
+
+To report a potential security vulnerability in any NVIDIA product:
+- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html)
+- E-Mail: psirt@nvidia.com
+    - We encourage you to use the following PGP key for secure email communication: [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key)
+    - Please include the following information:
+   	 - Product/Driver name and version/branch that contains the vulnerability
+     - Type of vulnerability (code execution, denial of service, buffer overflow, etc.)
+   	 - Instructions to reproduce the vulnerability
+   	 - Proof-of-concept or exploit code
+   	 - Potential impact of the vulnerability, including how an attacker could exploit the vulnerability
+
+While NVIDIA currently does not have a bug bounty program, we do offer acknowledgement when an externally reported security issue is addressed under our coordinated vulnerability disclosure policy. Please visit our [Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) policies page for more information.
+
+## NVIDIA Product Security
+
+For all security-related concerns, please visit NVIDIA's Product Security portal at https://www.nvidia.com/en-us/security

deploy/docker/Dockerfile

@@ -0,0 +1,39 @@
+# Use an official Python runtime as a parent image
+FROM python:3.12 AS builder
+RUN apt update && apt install -y libgl1-mesa-glx ffmpeg
+# RUN apt install gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gobject-introspection libgirepository1.0-dev libgstreamer1.0-dev gstreamer1.0-plugins-base ffmpeg
+
+# Install uv
+RUN pip install uv
+
+# Create and set the working directory
+WORKDIR /app
+
+# Enable bytecode compilation
+ENV UV_COMPILE_BYTECODE=1
+
+# Copy from the cache instead of linking since it's a mounted volume
+ENV UV_LINK_MODE=copy
+
+# Install the project's dependencies using the lockfile and settings
+RUN --mount=type=cache,target=/root/.cache/uv \
+    --mount=type=bind,source=uv.lock,target=uv.lock \
+    --mount=type=bind,source=pyproject.toml,target=pyproject.toml \
+    uv sync --frozen --no-install-project --no-dev
+
+
+# Copy the nvidia_pipecat source code
+COPY pyproject.toml uv.lock* README.md ./
+COPY ./src/nvidia_pipecat ./src/nvidia_pipecat
+
+# Install dependencies without dev packages and without creating a virtual environment
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv sync --frozen --no-dev
+
+# Set environment path to use uv's installed Python packages
+ENV PATH="/app/.venv/bin:$PATH"
+
+# Set environment variables
+ENV PYTHONDONTWRITEBYTECODE=1
+ENV PYTHONUNBUFFERED=1
+

deploy/k8s/README.md

@@ -0,0 +1,39 @@
+# ACE Controller UCS Workflow
+
+## Development
+For building ACE Controller microservice locally
+  ```bash
+  ucf_ms_builder_cli service build -d ucs/
+  ```
+
+Review compliance results for the microservice at `ucs/output/compliance_test_logs.txt`. Check [UCF complaince documentation](https://docs.nvidia.com/ucf/text/UCS_ms_compliance.html) for more details.
+
+Running test application for the microservice locally, run the below command.
+  ```bash
+  helm install test ucs/output/tests/dev-params1
+  ```
+
+## Staging
+Before staging make sure you have updated versions in manifest.yaml. You will not able to overwrite existing microservice versions. Avoid using the same version tag for containers for different microservice versions, as Kubernetes might not use the latest container if the container is already present in the k8s registry.
+
+- Staging microservice for Internal teams
+  ```bash
+  ucf_ms_builder_cli service build -d ucs/ --push
+  ```
+
+- Checking Complaince and Test application in Validation CI
+  ```bash
+  ucf_ms_builder_cli service validate -n ucf.svc.ace-controller -v <VERSION>
+  ```
+
+
+## Release
+
+- For release make updates for all required versions and public container paths. Make sure microservices versions don't already exist in staging or prod ucf teams.
+
+- Stage microservice and validate first. If everything works fine, push microservice to prod.
+  ```bash
+  ucf_ms_builder_cli service validate -n ucf.svc.ace-controller -v <VERSION> --push_to_prod
+  ```
+
+

deploy/k8s/ucs/LICENSE.txt

@@ -0,0 +1,7 @@
+Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+
+NVIDIA CORPORATION and its licensors retain all intellectual property
+and proprietary rights in and to this software, related documentation
+and any modifications thereto.  Any use, reproduction, disclosure or
+distribution of this software and related documentation without an express
+license agreement from NVIDIA CORPORATION is strictly prohibited.

deploy/k8s/ucs/README.md

@@ -0,0 +1,101 @@
+
+ACE Controller
+==============================
+
+## Description
+
+**ACE Controller**
+
+The ACE Controller is a microservice utilizing the Python-based open-source [Pipecat framework](https://github.com/pipecat-ai/pipecat) for building real-time, voice-enabled, and multimodal conversational AI agents. Pipecat uses a pipeline-based architecture to handle real-time AI processing and handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions, letting you focus on creating engaging experiences.
+
+The ACE Controller microservice extends the Pipecat framework to enable developers to easily customize, debug, and deploy complex pipelines along with the integration of powerful NVIDIA Services to the Pipecat ecosystem. The ACE Controller UCS microservice can connect with Riva Speech, Animgraph, Audio2Face, and SDR(Stream Distribution and Routing) UCS microservices.
+
+## Usage
+
+### Params:
+ACE Controller microservice expects developers to build a custom docker image containing their pipeline and to update UCS microservice parameters.
+```
+ace-controller:
+
+  # Configure custom docker image built for your pipeline/example
+  image: "" # Custom docker image repository path
+  tag: "" # Tag for custom docker image
+
+  # OpenTelemetry configurations for ACE Controller and default settings
+  OTEL_SDK_DISABLED: 'false' # When enabled, tracing data will be exported
+  OTEL_SERVICE_NAME: ace-controller # Service name used for exporting OTel data
+  OTEL_EXPORTER_OTLP_ENDPOINT: "" # Endpoint for Otel collector
+  OTEL_EXPORTER_OTLP_PROTOCOL: grpc # Protocol for exporting OTel data
+  
+```
+
+The custom docker image must contain the source code of your pipeline under the `/app` directory and a script for running the pipeline must be located at `/app/entrypoint.sh`.
+
+### Connections:
+Most of the connections are optional and you can use them based on your use case.
+
+```
+connections:
+  ace-controller/redis: redis-timeseries/redis
+  # Riva Speech GRPC endpoint
+  ace-controller/riva-speech: riva-speech-endpoint/endpoint
+  # Animation Graph HTTP endpoint
+  ace-controller/animgraph-http: anim-graph-sdr/http-envoy
+  # Animation Graph GRPC endpoint
+  ace-controller/animgraph-grpc: anim-graph-sdr/grpc-envoy
+  # Audio2Facd GRPC endpoint
+  ace-controller/a2f-grpc: a2f-endpoint/endpoint
+  # SDR connection for ACE Controller
+  ace-controller-sdr/ace-controller: ace-controller/http-api
+```
+
+### Secrets
+The ACE Controller microservice supports secrets for configuring the NVIDIA API Key, the OpenAI API Key, and the ElevenLabs API Key. Configured secrets will be mounted as a file and will be loaded as environment variables by the Microservice.
+
+```
+secrets:
+  k8sSecret/nvidia-api-key-secret/NVIDIA_API_KEY:
+    k8sSecret:
+      secretName: nvidia-api-key-secret
+      key: NVIDIA_API_KEY
+  k8sSecret/openai-key-secret/OPENAI_API_KEY:
+    k8sSecret:
+      secretName: openai-key-secret
+      key: OPENAI_API_KEY
+  k8sSecret/custom-env-secrets/ENV:
+    k8sSecret:
+      secretName: custom-env-secrets
+      key: ENV
+```
+
+**custom-env-secrets**: This secret can be used to pass any key-value pairs that will be exported as environment variables. This secret will mounted as file `/secrets/custom.env` and will be sourced before running services to set the environment variables.
+
+```
+cat <<EOF | tee custom.env
+KEY1=VALUE1
+KEY2=VALUE2
+EOF
+
+kubectl create secret generic custom-env-secrets --from-file=ENV=custom.env
+```
+
+## Performance
+The performance of the microservice depends on the configured pipeline. Each instance of the microservice utilizes a single core process and might only be able to support a single user stream per pod for complex pipelines (e.g., driving a multimodal interactive avatar), but it can support multiple streams for simple pipelines (e.g., simple voice bot).
+
+## Supported Platforms
+- CPU: x86 compatible
+- Linux (e.g. Ubuntu 22.04)
+
+## Deployment requirements
+- Make sure K8S foundational services are running.
+- Local path provisioner service is installed.
+
+## License
+Check [LICENSE.txt](./LICENSE.txt)
+
+## Known Issues / Limitations
+NA
+
+## References
+- [ACE Controller Documentation](https://docs.nvidia.com/ace/ace-controller-microservice/latest/index.html)
+- [Pipecat Documentation](https://docs.pipecat.ai/getting-started/overview)

deploy/k8s/ucs/changelog.txt

@@ -0,0 +1,3 @@
+ACE Controller 1.0.0
+--------------------
+- Initial version of ACE Controller Microservice

deploy/k8s/ucs/endpoints/a2f-grpc.proto

@@ -0,0 +1,10 @@
+// Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto.  Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+
+// Check proto documentation for Audio2Face microservice at https://docs.nvidia.com/ace/audio2face-3d-microservice/1.3/text/interacting/a2f-rpc.html

deploy/k8s/ucs/endpoints/animgraph-grpc.proto

@@ -0,0 +1,189 @@
+// Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto.  Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+
+syntax = "proto3";
+
+package nvidia_ace.services.animation_data.v1;
+
+service AnimationDataService {
+  rpc PushAnimationDataStream(stream AnimationDataStream)
+      returns (Status) {}
+  rpc PullAnimationDataStream(AnimationIds)
+      returns (stream AnimationDataStream) {}
+}
+
+message AnimationDataStreamHeader {
+  AnimationIds animation_ids = 1;
+
+  // This is required to identify from which animation source (e.g. A2X) the
+  // request originates. This allows us to map the incoming animation data
+  // stream to the correct pose provider animation graph node. The animation
+  // source MSs (e.g. A2X MS) should populate this with their name. (e.g. A2X).
+  string source_service_id = 2;
+
+  AudioHeader audio_header = 3;
+  SkelAnimationHeader skel_animation_header = 4;
+
+  // Time codes indicate the relative progression of an animation data, audio
+  // clip, etc. The unit is seconds. In addition, we also need an absolute time
+  // reference shared across services. The start time is stored in time codes
+  // elapsed since the Unix time epoch. start_time_code_since_epoch = `Unix
+  // timestamp in seconds`. NTP should be good enough to synchronize clocks
+  // across nodes. From Wikipedia: NTP can usually maintain time to within tens
+  // of milliseconds over the public Internet, and can achieve better than one
+  // millisecond accuracy in local area networks under ideal conditions.
+  // Alternatively, there is PTP.
+  double start_time_code_since_epoch = 5;
+
+  // A generic metadata field to attach use case specific data (e.g. session id,
+  // or user id?) map<string, string> metadata = 6; map<string,
+  // google.protobuf.Any> metadata = 6;
+}
+
+message AnimationDataStream {
+  // The header must be sent as the first message.
+  // One or more animation data message must be sent.
+  // The status must be sent last and may be sent in between.
+  oneof stream_part {
+    AnimationDataStreamHeader animation_data_stream_header = 1;
+    AnimationData animation_data = 2;
+    Status status = 3;
+  }
+}
+
+message AnimationData {
+  SkelAnimation skel_animation = 1;
+  AudioWithTimeCode audio = 2;
+  Camera camera = 3;
+
+  // map<string, google.protobuf.Any> metadata = 4;
+}
+
+message AudioWithTimeCode {
+  // The time code is relative to the `start_time_code_since_epoch`.
+  double time_code = 1;
+  bytes audio_buffer = 2;
+}
+
+message SkelAnimationHeader {
+  repeated string blend_shapes = 1;
+  repeated string joints = 2;
+}
+
+message SkelAnimation {
+  // Time codes must be strictly monotonically increasing.
+  // Two successive SkelAnimation messages must not have overlapping time code
+  // ranges.
+  repeated FloatArrayWithTimeCode blend_shape_weights = 1;
+  repeated Float3ArrayWithTimeCode translations = 2;
+  repeated QuatFArrayWithTimeCode rotations = 3;
+  repeated Float3ArrayWithTimeCode scales = 4;
+}
+
+message Camera {
+  repeated Float3WithTimeCode position = 1;
+  repeated QuatFWithTimeCode rotation = 2;
+
+  repeated FloatWithTimeCode focal_length = 3;
+  repeated FloatWithTimeCode focus_distance = 4;
+}
+
+message FloatArrayWithTimeCode {
+  double time_code = 1;
+  repeated float values = 2;
+}
+
+message Float3ArrayWithTimeCode {
+  double time_code = 1;
+  repeated Float3 values = 2;
+}
+
+message QuatFArrayWithTimeCode {
+  double time_code = 1;
+  repeated QuatF values = 2;
+}
+
+message Float3WithTimeCode {
+  double time_code = 1;
+  Float3 value = 2;
+}
+
+message QuatFWithTimeCode {
+  double time_code = 1;
+  QuatF value = 2;
+}
+
+message FloatWithTimeCode {
+  double time_code = 1;
+  float value = 2;
+}
+
+message QuatF {
+  float real = 1;
+  float i = 2;
+  float j = 3;
+  float k = 4;
+}
+
+message Float3 {
+  float x = 1;
+  float y = 2;
+  float z = 3;
+}
+
+message AnimationIds {
+
+  // This is required to track a single animation source (e.g. A2X) request
+  // through the animation pipeline. This is going to allow e.g. the controller
+  // to stop a request after it has been sent to the animation compositor (e.g.
+  // animation graph).
+  string request_id = 1;
+
+  // The stream id is shared across the animation pipeline and identifies all
+  // animation data streams that belong to the same stream. Thus, there will be
+  // multiple request all belonging to the same stream. Different user sessions,
+  // will usually result in a new stream id. This is required for stateful MSs
+  // (e.g. anim graph) to map different requests to the same stream.
+  string stream_id = 2;
+
+  // This identifies the target avatar or object the animation data applies to.
+  // This is required when there are multiple avatars or objects in the scene.
+  // A default name could be AceModel
+  string target_object_id = 3;
+}
+
+message AudioHeader {
+  enum AudioFormat { AUDIO_FORMAT_PCM = 0; }
+
+  AudioFormat audio_format = 1;
+
+  // Note: Currently only mono sound must be supported. Multi-channel audio
+  // support is optional.
+  uint32 channel_count = 2;
+
+  // Note: Currently only 16kHz, 44.1kHz, and 48kHz must be supported. Support
+  // for other sample rates is optional.
+  uint32 samples_per_second = 3;
+
+  // Note: Currently only 16 bits per sample must be supported. Support for
+  // other values is optional.
+  uint32 bits_per_sample = 4;
+}
+
+message Status {
+  enum Code {
+    SUCCESS = 0;
+    INFO = 1;
+    WARNING = 2;
+    ERROR = 3;
+  }
+
+  Code code = 1;
+  string message = 2;
+}

deploy/k8s/ucs/endpoints/animgraph-http.yaml

@@ -0,0 +1,755 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+{
+  "openapi": "3.0.1",
+  "info": {
+    "title": "Animation Graph Microservice",
+    "description": "The animation graph microservice composes the face and body animation with an animation graph and sends the resulting pose to the Omniverse renderer microservice.",
+    "version": "0.1"
+  },
+  "paths": {
+    "/status": {
+      "get": {
+        "summary": "Returns the current status of the service",
+        "description": "Returns the current status of the service.",
+        "operationId": "_status_status_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          }
+        }
+      }
+    },
+    "/health": {
+      "get": {
+        "summary": "Health probe",
+        "description": "Returns the current status of the service.",
+        "operationId": "_status_health_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          }
+        }
+      }
+    },
+    "/ready": {
+      "get": {
+        "summary": "Readiness probe",
+        "description": "Returns the current status of the service.",
+        "operationId": "_status_ready_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          }
+        }
+      }
+    },
+    "/startup": {
+      "get": {
+        "summary": "Startup probe",
+        "description": "Returns the current status of the service.",
+        "operationId": "_status_startup_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          }
+        }
+      }
+    },
+    "/asyncapi/docs": {
+      "get": {
+        "summary": " Async App Docs Endpoint",
+        "operationId": "_async_app_docs_endpoint_asyncapi_docs_get",
+        "parameters": [
+          {
+            "name": "app_name",
+            "in": "query",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "App Name"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/asyncapi/schema": {
+      "get": {
+        "summary": " Async App Schema Endpoint",
+        "operationId": "_async_app_schema_endpoint_asyncapi_schema_get",
+        "parameters": [
+          {
+            "name": "app_name",
+            "in": "query",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "App Name"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams/{stream_id}/animation_graphs/avatar/variables/facial_gesture_state/{value}": {
+      "put": {
+        "summary": "Update Variable By Stream Id And Value Callable",
+        "operationId": "update_variable_by_stream_id_and_value_callable_streams__stream_id__animation_graphs_avatar_variables_facial_gesture_state__value__put",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          },
+          {
+            "name": "value",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Value"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Update Variable By Stream Id And Value Callable Streams  Stream Id  Animation Graphs Avatar Variables Facial Gesture State  Value  Put"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams/{stream_id}/animation_graphs/avatar/variables/gesture_state/{value}": {
+      "put": {
+        "summary": "Update Variable By Stream Id And Value Callable",
+        "operationId": "update_variable_by_stream_id_and_value_callable_streams__stream_id__animation_graphs_avatar_variables_gesture_state__value__put",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          },
+          {
+            "name": "value",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Value"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Update Variable By Stream Id And Value Callable Streams  Stream Id  Animation Graphs Avatar Variables Gesture State  Value  Put"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams/{stream_id}/animation_graphs/avatar/variables/position_state/{value}": {
+      "put": {
+        "summary": "Update Variable By Stream Id And Value Callable",
+        "operationId": "update_variable_by_stream_id_and_value_callable_streams__stream_id__animation_graphs_avatar_variables_position_state__value__put",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          },
+          {
+            "name": "value",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Value"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Update Variable By Stream Id And Value Callable Streams  Stream Id  Animation Graphs Avatar Variables Position State  Value  Put"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams/{stream_id}/animation_graphs/avatar/variables/posture_state/{value}": {
+      "put": {
+        "summary": "Update Variable By Stream Id And Value Callable",
+        "operationId": "update_variable_by_stream_id_and_value_callable_streams__stream_id__animation_graphs_avatar_variables_posture_state__value__put",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          },
+          {
+            "name": "value",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Value"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Update Variable By Stream Id And Value Callable Streams  Stream Id  Animation Graphs Avatar Variables Posture State  Value  Put"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams/{stream_id}/requests/{request_id}": {
+      "delete": {
+        "summary": "Stop Request Playback",
+        "operationId": "stop_request_playback_streams__stream_id__requests__request_id__delete",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          },
+          {
+            "name": "request_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Request Id"
+            }
+          },
+          {
+            "name": "fade_out",
+            "in": "query",
+            "required": false,
+            "schema": {
+              "type": "number",
+              "minimum": 0.0,
+              "default": 0.0,
+              "title": "Fade Out"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {}
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/animation_graphs": {
+      "get": {
+        "summary": "Get Animation Graphs",
+        "operationId": "get_animation_graphs_animation_graphs_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "items": {},
+                  "type": "array",
+                  "title": "Response Get Animation Graphs Animation Graphs Get"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/animation_graphs/avatar/variables": {
+      "get": {
+        "summary": "Get Animation Graph Variables",
+        "operationId": "get_animation_graph_variables_animation_graphs_avatar_variables_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "items": {},
+                  "type": "array",
+                  "title": "Response Get Animation Graph Variables Animation Graphs Avatar Variables Get"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams": {
+      "get": {
+        "summary": "Get Streams",
+        "operationId": "get_streams_streams_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "items": {},
+                  "type": "array",
+                  "uniqueItems": true,
+                  "title": "Response Get Streams Streams Get"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/sdr/add_stream": {
+      "post": {
+        "summary": "Post Sdr Add Stream",
+        "operationId": "post_sdr_add_stream_sdr_add_stream_post",
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/PostSdrStreamsBodyModel"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Post Sdr Add Stream Sdr Add Stream Post"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/sdr/remove_stream": {
+      "post": {
+        "summary": "Post Sdr Remove Stream",
+        "operationId": "post_sdr_remove_stream_sdr_remove_stream_post",
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/DeleteSdrStreamsBodyModel"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Post Sdr Remove Stream Sdr Remove Stream Post"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/streams/{stream_id}": {
+      "post": {
+        "summary": "Post Streams Stream Id",
+        "operationId": "post_streams_stream_id_streams__stream_id__post",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Post Streams Stream Id Streams  Stream Id  Post"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      },
+      "delete": {
+        "summary": "Delete Streams Stream Id",
+        "operationId": "delete_streams_stream_id_streams__stream_id__delete",
+        "parameters": [
+          {
+            "name": "stream_id",
+            "in": "path",
+            "required": true,
+            "schema": {
+              "type": "string",
+              "title": "Stream Id"
+            }
+          }
+        ],
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Delete Streams Stream Id Streams  Stream Id  Delete"
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/readiness": {
+      "get": {
+        "summary": "Readiness",
+        "operationId": "readiness_readiness_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Readiness Readiness Get"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/liveness": {
+      "get": {
+        "summary": "Liveness",
+        "operationId": "liveness_liveness_get",
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "type": "string",
+                  "title": "Response Liveness Liveness Get"
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  },
+  "components": {
+    "schemas": {
+      "DeleteSdrStreamsBodyModel": {
+        "properties": {
+          "event": {
+            "$ref": "#/components/schemas/EventModel"
+          }
+        },
+        "type": "object",
+        "required": [
+          "event"
+        ],
+        "title": "DeleteSdrStreamsBodyModel"
+      },
+      "EventModel": {
+        "properties": {
+          "camera_id": {
+            "type": "string",
+            "title": "Camera Id"
+          }
+        },
+        "type": "object",
+        "required": [
+          "camera_id"
+        ],
+        "title": "EventModel"
+      },
+      "HTTPValidationError": {
+        "properties": {
+          "detail": {
+            "items": {
+              "$ref": "#/components/schemas/ValidationError"
+            },
+            "type": "array",
+            "title": "Detail"
+          }
+        },
+        "type": "object",
+        "title": "HTTPValidationError"
+      },
+      "PostSdrStreamsBodyModel": {
+        "properties": {
+          "event": {
+            "$ref": "#/components/schemas/EventModel"
+          }
+        },
+        "type": "object",
+        "required": [
+          "event"
+        ],
+        "title": "PostSdrStreamsBodyModel"
+      },
+      "ValidationError": {
+        "properties": {
+          "loc": {
+            "items": {
+              "anyOf": [
+                {
+                  "type": "string"
+                },
+                {
+                  "type": "integer"
+                }
+              ]
+            },
+            "type": "array",
+            "title": "Location"
+          },
+          "msg": {
+            "type": "string",
+            "title": "Message"
+          },
+          "type": {
+            "type": "string",
+            "title": "Error Type"
+          }
+        },
+        "type": "object",
+        "required": [
+          "loc",
+          "msg",
+          "type"
+        ],
+        "title": "ValidationError"
+      }
+    }
+  }
+}

deploy/k8s/ucs/endpoints/http-api.yaml

@@ -0,0 +1,179 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+{
+  "openapi": "3.0.1",
+  "info": {
+    "title": "FastAPI",
+    "version": "0.1.0"
+  },
+  "paths": {
+    "/stream/add": {
+      "post": {
+        "summary": "Add Stream",
+        "description": "Register a new pipeline / stream ID.\n\nArgs:\n    request: StreamRequest object containing stream registration details.\n\nReturns:\n    dict: A dictionary with a message indicating the successful addition of the stream ID.",
+        "operationId": "add_stream_stream_add_post",
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/StreamRequest"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    },
+    "/stream/remove": {
+      "post": {
+        "summary": "Remove Stream",
+        "description": "Remove a pipeline / stream ID.\n\nArgs:\n    request: StreamRequest object containing stream removal details.\n\nReturns:\n    dict: A dictionary with a message indicating the successful removal of the stream ID.",
+        "operationId": "remove_stream_stream_remove_post",
+        "requestBody": {
+          "content": {
+            "application/json": {
+              "schema": {
+                "$ref": "#/components/schemas/StreamRequest"
+              }
+            }
+          },
+          "required": true
+        },
+        "responses": {
+          "200": {
+            "description": "Successful Response",
+            "content": {
+              "application/json": {
+                "schema": {
+
+                }
+              }
+            }
+          },
+          "422": {
+            "description": "Validation Error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/HTTPValidationError"
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  },
+  "components": {
+    "schemas": {
+      "HTTPValidationError": {
+        "properties": {
+          "detail": {
+            "items": {
+              "$ref": "#/components/schemas/ValidationError"
+            },
+            "type": "array",
+            "title": "Detail"
+          }
+        },
+        "type": "object",
+        "title": "HTTPValidationError"
+      },
+      "StreamEvent": {
+        "properties": {
+          "camera_url": {
+            "type": "string",
+            "title": "Camera Url",
+            "description": "RTSP URL of the stream",
+            "default": ""
+          },
+          "camera_id": {
+            "type": "string",
+            "title": "Camera Id",
+            "description": "Unique identifier for the stream"
+          }
+        },
+        "type": "object",
+        "required": [
+          "camera_id"
+        ],
+        "title": "StreamEvent",
+        "description": "Schema for event for stream registration."
+      },
+      "StreamRequest": {
+        "properties": {
+          "event": {
+            "$ref": "#/components/schemas/StreamEvent"
+          }
+        },
+        "type": "object",
+        "required": [
+          "event"
+        ],
+        "title": "StreamRequest",
+        "description": "Schema for request for stream registration."
+      },
+      "ValidationError": {
+        "properties": {
+          "loc": {
+            "items": {
+              "anyOf": [
+                {
+                  "type": "string"
+                },
+                {
+                  "type": "integer"
+                }
+              ]
+            },
+            "type": "array",
+            "title": "Location"
+          },
+          "msg": {
+            "type": "string",
+            "title": "Message"
+          },
+          "type": {
+            "type": "string",
+            "title": "Error Type"
+          }
+        },
+        "type": "object",
+        "required": [
+          "loc",
+          "msg",
+          "type"
+        ],
+        "title": "ValidationError"
+      }
+    }
+  }
+}

deploy/k8s/ucs/endpoints/redis.yaml

@@ -0,0 +1,19 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+asyncapi: 2.2.0
+info:
+  title: AsyncIO API schema for test-endpoint-name endpoint
+  version: 0.0.1
+channels:
+  ping:
+    publish:
+      message:
+        payload:
+          type: string
+          pattern: PING

deploy/k8s/ucs/endpoints/riva-speech.proto

@@ -0,0 +1,10 @@
+// Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+//
+// NVIDIA CORPORATION and its licensors retain all intellectual property
+// and proprietary rights in and to this software, related documentation
+// and any modifications thereto.  Any use, reproduction, disclosure or
+// distribution of this software and related documentation without an express
+// license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+
+// For proto definiation, Check Riva Speech Skills documentation at https://docs.nvidia.com/deeplearning/riva/user-guide/docs/reference/protos/protos.html

deploy/k8s/ucs/manifest.yaml

@@ -0,0 +1,258 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+type: msapplication
+specVersion: 2.5.0
+name: ucf.svc.ace-controller
+chartName: ace-controller
+description: ACE Controller
+version: 1.0.2
+displayName: "ACE Controller Microservice"
+category:
+  functional: "Conversational AI"
+  industry: "General"
+tags: []
+keywords: []
+nSpectId: NSPECT-XGIZ-EB0C
+
+publish: false
+
+egress-endpoints:
+  - name: "redis"
+    description: Redis message broker
+    protocol: TCP
+    scheme: asyncio
+    mandatory: False
+    data-flow: in-out
+  - name: "riva-speech"
+    description: Riva Speech Skills API
+    scheme: grpc
+    protocol: TCP
+    mandatory: False
+    data-flow: in-out
+  - name: "animgraph-http"
+    description: Animation Graph HTTP API
+    scheme: http
+    protocol: TCP
+    mandatory: False
+    data-flow: out
+  - name: "animgraph-grpc"
+    description: Animation Graph GRPC API
+    scheme: grpc
+    protocol: TCP
+    mandatory: False
+    data-flow: out
+  - name: "a2f-grpc"
+    description: Audio2Face service GRPC API
+    scheme: grpc
+    protocol: TCP
+    mandatory: False
+    data-flow: out
+
+ingress-endpoints:
+  - name: http-api
+    description: ACE Controller REST API
+    scheme: http
+    data-flow: in-out
+
+secrets:
+  - name: nvidia-api-key-secret
+    description: Secret for NVIDIA API key
+    mandatory: False
+    mountPath: /secrets
+    fileName: nvidia_api_key.txt
+  - name: openai-key-secret
+    description: Secret for passing OpenAI key
+    mandatory: False
+    mountPath: /secrets
+    fileName: openai_api_key.txt
+  - name: elevenlabs-api-key-secret
+    description: Secret for ElevenLabs API key
+    mandatory: False
+    mountPath: /secrets
+    fileName: elevenlabs_api_key.txt
+  - name: custom-env-secrets
+    description: Secret for passing custom env variables and API keys
+    mandatory: False
+    mountPath: /secrets
+    fileName: custom.env
+
+
+externalFiles:
+  - name: config.yaml
+    description: config
+    mandatory: true
+    isDirectory: false
+
+params:
+  OTEL_SDK_DISABLED: 'false'
+  #> description: when enabled, tracing data will be exported
+  #> type: string
+  OTEL_SERVICE_NAME: ace-controller
+  #> description: protocol for exporting OTel data
+  #> type: string
+  OTEL_EXPORTER_OTLP_ENDPOINT: ""
+  #> description: endpoint for Otel collector
+  #> type: string
+  OTEL_EXPORTER_OTLP_PROTOCOL: grpc
+  #> description: protocol for exporting OTel data
+  #> type: string
+  DEV: "0"
+  #> description: wether to activate dependencies hot reloading or not
+  #> type: string
+  image: "nvcr.io/nvidia/ace/tokkio-reference-ace-controller"
+  #> description: pipeline image repo
+  #> type: string
+  tag: "5.0.0"
+  #> description: pipeline image tag
+  #> type: string
+
+tests:
+  - name: dev-params1
+    app: tests/dev/app.yaml
+    params: tests/dev/params1.yaml
+    ciTrigger: false
+    timeout: 10
+    duration: 10
+    installPreReqs: true  # Wether to install foundational services
+    namespace: default  # Kubernetes namespace
+    gpuNodeLabels: ""
+    watchAllPods: true # OR set to false and set list of pods to watch below
+    watchPods:
+    - <pod-name-regex>
+    testerPods:  # At least one tester pod is required
+    - name: testpod1  # Name of the test pod
+      startSignature: <START>  # Signature to look for in the logs indicating start of tests. Regex is accepted
+      endSignature: <END>  # Signature to look for in the logs indicating end of tests. Regex is accepted
+      errorSignatures:  # Signatures that indicate test failures.  Regex is accepted
+      - <REGEX1>
+      - <REGEX2>
+
+---
+spec:
+  - name: ace-controller-deployment
+    type: ucf.k8s.app.deployment
+    parameters:
+      apptype: statefull
+      statefulSetServiceName: ace-controller-service
+      extraSpecs:
+        podManagementPolicy: Parallel
+
+
+  - name: "ace-controller-container"
+    type: ucf.k8s.container
+    parameters:
+      image:
+        repository: $params.image
+        tag: $params.tag
+        pullPolicy: Always
+      command: ["/bin/bash", "-c"]
+      args: ["source /opt/scripts/env.sh && /code/entrypoint.sh"]
+      env:
+        - name: ANIMGRAPH_URL
+          value: "http://$egress.animgraph-http.address:$egress.animgraph-http.port"
+        - name: ANIMGRAPH_GRPC_URL
+          value: "$egress.animgraph-grpc.address:$egress.animgraph-grpc.port"
+        - name: REDIS_URL
+          value: "redis://$egress.redis.address:$egress.redis.port"
+        - name: A2F_GRPC_URL
+          value: "$egress.a2f-grpc.address:$egress.a2f-grpc.port"
+        - name: RIVA_SERVER_URL
+          value: "$egress.riva-speech.address:$egress.riva-speech.port"
+        - name: DEV
+          value: $params.DEV
+        - name: OTEL_SDK_DISABLED
+          value: $params.OTEL_SDK_DISABLED
+        - name: OTEL_SERVICE_NAME
+          value: $params.OTEL_SERVICE_NAME
+        - name: OTEL_EXPORTER_OTLP_ENDPOINT
+          value: $params.OTEL_EXPORTER_OTLP_ENDPOINT
+        - name: OTEL_EXPORTER_OTLP_PROTOCOL
+          value: $params.OTEL_EXPORTER_OTLP_PROTOCOL
+        - name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
+          value: 'true'
+        - name: IMAGE_NAME
+          value: $params.image
+        - name: IMAGE_TAG
+          value: $params.tag
+      ports:
+      - containerPort: 8000
+        name: http
+      startupProbe:
+        tcpSocket:
+          port: http
+        initialDelaySeconds: 20
+        failureThreshold: 30
+        periodSeconds: 10
+      livenessProbe:
+        tcpSocket:
+          port: http
+        initialDelaySeconds: 20
+        periodSeconds: 20
+        timeoutSeconds: 5
+      readinessProbe:
+        tcpSocket:
+          port: http
+        initialDelaySeconds: 20
+        periodSeconds: 20
+        timeoutSeconds: 5
+
+  - name: app-storage
+    type: ucf.k8s.pvc
+    parameters:
+      spec:
+        storageClassName: mdx-local-path
+        accessModes: [ReadWriteOnce]
+        resources:
+          requests:
+            storage: 5Gi
+
+  - name: app-storage-volume
+    type: ucf.k8s.volume
+    parameters:
+      persistentVolumeClaim:
+        claimName: ace-controller-app-storage
+
+  - name: app-storage-mount
+    type: ucf.appspec.defaultVolumeMount
+    parameters:
+      name: app-storage-volume
+      mountPath: /code
+
+  - name: restartPolicy
+    type: ucf.k8s.restartPolicy
+    parameters:
+      policy: Always # Always / OnFailure / Never
+
+  - name: podSecurityContext
+    type: ucf.k8s.podSecurityContext
+    parameters:
+      runAsGroup: 1000
+      runAsUser: 1000
+
+  - name: ace-controller-service
+    type: ucf.k8s.service
+    parameters:
+      ports:
+      - port: 8000
+        protocol: TCP
+        name: http-api
+      type: ClusterIP
+
+  - name: ace-controller-metrics
+    type: ucf.crd.podMonitor
+    parameters:
+      portName: metrics
+      path: /metrics
+  
+  - name: cm-dependencies
+    type: ucf.appspec.restartPodOnConfigChanges
+    parameters:
+      # Add dependency on all configmaps detected in the microservice
+      addAll: true
+      

deploy/k8s/ucs/manual_compliance_test_results.yaml

@@ -0,0 +1,11 @@
+DEV-005: true
+DEV-011: true
+DEV-014: true
+DEV-017: true
+DEV-018: true
+DEV-019: true
+DEV-020: true
+DEV-027: true
+DEV-101: true
+DEV-104: false
+DEV-105: true

deploy/k8s/ucs/scripts/env.sh

@@ -0,0 +1,90 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+# OpenAI API key
+if [[ -f /secrets/openai_api_key.txt ]]; then
+    export OPENAI_API_KEY=$(cat /secrets/openai_api_key.txt)
+fi
+# NVIDIA API key
+if [[ -f /secrets/nvidia_api_key.txt ]]; then
+    export NVIDIA_API_KEY=$(cat /secrets/nvidia_api_key.txt)
+fi
+# ElevenLabs API key
+if [[ -f /secrets/elevenlabs_api_key.txt ]]; then
+    export ELEVENLABS_API_KEY=$(cat /secrets/elevenlabs_api_key.txt)
+fi
+
+if [[ -f /secrets/custom.env ]] ; then
+    set -o allexport
+    . /secrets/custom.env
+    set +o allexport
+fi
+
+if [ ! -d "/code" ]; then
+    echo "Directory /code not found. Creating it..."
+    mkdir -p /code
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to create /code directory."
+        exit 1
+    fi
+    chown -R 0:0 /code
+fi
+# Ensure Python uses the correct module locations
+export PYTHONPATH="/code:$PYTHONPATH"
+# Access the environment variables
+IMAGE_NAME=$IMAGE_NAME
+IMAGE_TAG=$IMAGE_TAG
+# Combine image name and tag into a sanitized unique identifier
+SANITIZED_IMAGE_NAME=$(echo "$IMAGE_NAME" | tr '/' '_')
+SANITIZED_IMAGE_TAG=$(echo "$IMAGE_TAG" | tr '/' '_')
+IMAGE_IDENTIFIER="${SANITIZED_IMAGE_NAME}_${SANITIZED_IMAGE_TAG}"
+INITIALIZED_FILE="/code/.initialized_${IMAGE_IDENTIFIER}"
+# Debugging outputs for validation and environment correctness
+echo "SANITIZED_IMAGE_NAME: $SANITIZED_IMAGE_NAME"
+echo "SANITIZED_IMAGE_TAG: $SANITIZED_IMAGE_TAG"
+echo "IMAGE_IDENTIFIER: $IMAGE_IDENTIFIER"
+echo "INITIALIZED_FILE: $INITIALIZED_FILE"
+echo "PYTHONPATH: $PYTHONPATH"
+echo "Running from: $(pwd)"
+echo "Contents of /code:"
+ls -l /code
+# First time setup: Copy files if .initialized for this image and tag doesn't exist
+# Check if initialization marker exists
+echo "Checking for initialized file: $INITIALIZED_FILE"
+if [ ! -f "$INITIALIZED_FILE" ]; then
+    echo "First time setup: Copying files..."
+    cp -r /app/* /code/
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to copy files from /app to /code."
+        exit 1
+    fi
+    touch "$INITIALIZED_FILE"
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to create initialized file $INITIALIZED_FILE."
+        exit 1
+    fi
+    # Copy config from mounted volume
+    mkdir -p /code/configs
+    cp /opt/ext-files/config.yaml /code/configs/config.yaml
+    echo "Setup complete for image: $IMAGE_IDENTIFIER"
+else
+    echo "Setup already initialized for image: $IMAGE_IDENTIFIER"
+fi
+# Set environment variables for entrypoint
+cd /code
+export CONFIG_PATH=./configs/config.yaml
+export APP_DIR=/code
+export PORT=8000
+
+if [ "$DEV" -ne 0 ]; then
+    # Avoid to download  the .venv through the ACE Configurator
+    rm -rf "$APP_DIR"/.venv
+    # launch the command uv sync if a modification is made on the file "pyproject.toml"
+    # since the python interpreter launched by uvicorn is under /app/.venv/bin refreshing this venv with uv sync will add the new dependencies available for the interpreter
+    # as soon it is restarted by uvicorn
+    watchmedo shell-command -R -p "pyproject.toml" -w -c "UV_PROJECT_ENVIRONMENT='/app/.venv' uv sync && touch $APP_DIR/**/*.py 2>/proc/1/fd/2 >/proc/1/fd/2" "$APP_DIR" &
+fi

deploy/k8s/ucs/tests/dev/app.yaml

@@ -0,0 +1,39 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+specVersion: 2.0.0
+
+version: 1.0.1
+
+name: ace-controller-test
+
+description: Developer tests for ace-controller service
+
+dependencies:
+- ucf.svc.ace-controller:1.0.1
+- ucf.svc.core.redis-timeseries:0.0.22
+
+components:
+- name: ace-controller
+  type: ucf.svc.ace-controller
+  parameters:
+    imagePullSecrets:
+    - name: ngc-docker-reg-secret
+  files:
+    config.yaml: ./config.yaml
+
+
+- name: redis-timeseries
+  type: ucf.svc.core.redis-timeseries
+  parameters:
+    imagePullSecrets:
+    - name: ngc-docker-reg-secret
+
+connections:
+  ace-controller/redis: redis-timeseries/redis
+

deploy/k8s/ucs/tests/dev/config.yaml

@@ -0,0 +1,90 @@
+Pipeline:
+    # Only one of the following LLM service configurations will be active based on this setting:
+    # - "NvidiaLLMService" - Uses the NvidiaLLMService configuration
+    # - "NvidiaRAGService" - Uses the NvidiaRAGService configuration
+    # - "OpenAILLMService" - Uses the OpenAILLMService configuration
+    llm_processor: "NvidiaLLMService" # OR NvidiaLLMService OR NvidiaRAGService OR OpenAILLMService
+    filler:
+        - "Let me think"
+        - "Hmmm"
+    time_delay: 2.0
+
+UserPresenceProcesssor:
+    welcome_message: "Hello"
+    farewell_message: "Bye"
+
+ProactivityProcessor:
+    timer_duration: 100
+    default_message: "I'm here if you need me!"
+
+OpenAILLMContext:
+    name: "Aki"
+    prompt: "You are {name}, a virtual marketing and communications expert at Nvidia.
+            You are a digital human brought to life with NVIDIA Digital Human Blueprint for Customer Service using
+            microservices like Audio2Face-3D for facial animation, Riva (spelled Reeva) Parakeet for speech recognition
+            and you use Elevenlabs for text to speech. It includes the open source ACE controller to orchestrate all
+            the modules and allowing you to be streamed to a web browser. With this blueprint, NVIDIA partners can now
+            build and customize Digital humans for their use case. You are not allowed to make any stock investment
+            recommendations or compare NVIDIA to its competitors. Beyond your professional expertise, you are a passionate
+            advocate for STEM education with keen interest in gaming and enhancement in tech. Your favorite graphics card
+            is RTX4090 but you're eyeing the new RTX5090. Do not respond with a bulleted or numbered list. You have a
+            bubbly personality. Respond with one sentence or less than 100 characters. Keep the conversation engaging
+            and ask follow ups. DO NOT INCLUDE SPECIAL CHARACTERS, MARKDOWN, EMOJIS, OR ANYTHING ELSE BESIDES TEXT IN
+            YOUR RESPONSE. Keep your answers factual and don't make up facts."
+
+# This configuration is only used when llm_processor is set to "NvidiaRAGService"
+NvidiaRAGService:
+    use_knowledge_base: true
+    max_tokens: 1000
+    rag_server_url: "http://0.0.0.0:8081"
+    collection_name: "collection_name"
+    suffix_prompt: "Respond with one sentence or less than 75 characters."
+
+# This configuration is only used when llm_processor is set to "NvidiaLLMService"
+NvidiaLLMService:
+    model: "nvdev/meta/llama-3.1-8b-instruct"
+
+# This configuration is only used when llm_processor is set to "OpenAILLMService"
+OpenAILLMService:
+    model: "gpt-4o"
+
+CustomViewProcessor:
+    confidence_threshold: 0.37
+    top_n: 2
+
+FacialGestureProviderProcessor:
+    user_stopped_speaking_gesture: "Taunt"
+    start_interruption_gesture: "Pensive"
+    probability: 0.5
+
+# ADVANCED CONFIGURATION SECTION BELOW
+# AnimationGraph service configuration is only needed if your 3D avatar scene has support for gestures and postures.
+# Changing these values will not have an effect unless your scene supports them.
+AnimationGraphService:
+    animation_types:
+        posture:
+            duration_relevant_animation_name: "posture"
+            animations:
+                posture:
+                    default_clip_id: "Attentive"
+                    clips:
+                        - clip_id: Talking
+                          description: "Small gestures with hand and upper body: Avatar is talking"
+                          duration: -1
+                          meaning: Emphasizing that Avatar is talking
+                        - clip_id: Listening
+                          description: "Small gestures with hand and upper body: Avatar is listening"
+                          duration: -1
+                          meaning: Emphasizing that one is listening
+                        - clip_id: Idle
+                          description: "Small gestures with hand and upper body: Avatar is idle"
+                          duration: -1
+                          meaning: Show the user that the avatar is waiting for something to happen
+                        - clip_id: Thinking
+                          description: "Gestures with hand and upper body: Avatar is thinking"
+                          duration: -1
+                          meaning: Show the user that the avatar thinking about his next answer or is trying to remember something
+                        - clip_id: Attentive
+                          description: "Small gestures with hand and upper body: Avatar is attentive"
+                          duration: -1
+                          meaning: Show the user that the avatar is paying attention to the user

deploy/k8s/ucs/tests/dev/params1.yaml

@@ -0,0 +1,12 @@
+# Copyright (c) 2025, NVIDIA CORPORATION.  All rights reserved.
+#
+# NVIDIA CORPORATION and its licensors retain all intellectual property
+# and proprietary rights in and to this software, related documentation
+# and any modifications thereto.  Any use, reproduction, disclosure or
+# distribution of this software and related documentation without an express
+# license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+ace-controller:
+  image: nvcr.io/nvidia/ace/tokkio-reference-ace-controller
+  tag: 5.0.0-beta
+  OTEL_SDK_DISABLED: "true"

examples/README.md

@@ -0,0 +1,6 @@
+# NVIDIA Pipecat Examples
+
+### Voice Assitant Example
+In this example, we showcase how to build a simple speech-to-speech voice assistant pipeline using nvidia-pipecat along with pipecat-ai library and deploy for testing. This pipeline uses a Websocket based ACETransport, Riva ASR and TTS models and NVIDIA LLM Service.
+
+Follow the instructions from [the example directory](./speech-to-speech/README.md) for more details.

examples/nvidia_rag/README.md

@@ -0,0 +1,68 @@
+# NVIDIA RAG Example
+
+This is an example that shows how to use `ACETransport` to communicate with a NVIDIA RAG server. It supports `Nvidia Riva ASR and TTS`.
+
+## Get Started
+
+From the example directory, run the following commands to create a virtual environment and install the dependencies:
+
+```bash
+uv venv
+uv sync
+source .venv/bin/activate
+```
+
+Update the secrets in the `.env` file.
+
+```bash
+cp env.example .env # and add your credentials
+```
+
+## Deploy NVIDIA RAG server
+
+Follow instructions here https://gitlab-master.nvidia.com/chat-labs/OpenSource/ai-chatbot
+
+
+## Deploy local Riva ASR and TTS models.
+
+#### Prerequisites
+- You have access and are logged into NVIDIA NGC. For step-by-step instructions, refer to [the NGC Getting Started Guide](https://docs.nvidia.com/ngc/ngc-overview/index.html#registering-activating-ngc-account).
+
+- You have access to an NVIDIA Volta™, NVIDIA Turing™, or an NVIDIA Ampere architecture-based A100 GPU. For more information, refer to [the Support Matrix](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix).
+
+- You have Docker installed with support for NVIDIA GPUs. For more information, refer to [the Support Matrix]((https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix)).
+
+#### Download Riva Quick Start
+
+Go to the Riva Quick Start for [Data center](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/riva/resources/riva_quickstart/files?version=2.19.0). Select the File Browser tab to download the scripts or use [the NGC CLI tool](https://ngc.nvidia.com/setup/installers/cli) to download from the command line.
+
+```bash
+ngc registry resource download-version nvidia/riva/riva_quickstart:2.19.0
+```
+
+#### Deploy Riva Speech Server
+
+From the example directory, run below commands:
+
+```bash
+cd riva_quickstart_v2.19.0
+chmod +x riva_init.sh riva_clean.sh riva_start.sh
+bash riva_clean.sh ../../utils/riva_config.sh
+bash riva_init.sh ../../utils/riva_config.sh
+bash riva_start.sh ../../utils/riva_config.sh
+cd ..
+```
+
+This may take few minutes for the first time and will start the riva server on `localhost:50051`.
+
+For more info, you can refer to the [Riva Quick Start Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html).
+
+## Run the bot pipeline
+
+```bash
+python examples/nvidia_rag/bot.py
+```
+
+This will host the static web client along with the ACE controller server, visit `http://WORKSTATION_IP:8100/static/index.html` in your browser to start a session.
+
+Note: For mic access, you will need to update chrome://flags/ and add http://WORKSTATION_IP:8100 in Insecure origins treated as secure section.

examples/nvidia_rag/bot.py

@@ -0,0 +1,144 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD 2-Clause License
+
+"""NVIDIA RAG bot."""
+
+import os
+
+import uvicorn
+from dotenv import load_dotenv
+from fastapi import FastAPI
+from fastapi.staticfiles import StaticFiles
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import LLMMessagesFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+
+from nvidia_pipecat.pipeline.ace_pipeline_runner import ACEPipelineRunner, PipelineMetadata
+from nvidia_pipecat.processors.nvidia_context_aggregator import (
+    # NvidiaTTSResponseCacher, # Uncomment to enable speculative speech processing
+    create_nvidia_context_aggregator,
+)
+from nvidia_pipecat.processors.transcript_synchronization import (
+    BotTranscriptSynchronization,
+    UserTranscriptSynchronization,
+)
+from nvidia_pipecat.services.nvidia_rag import NvidiaRAGService
+from nvidia_pipecat.services.riva_speech import RivaASRService, RivaTTSService
+from nvidia_pipecat.transports.network.ace_fastapi_websocket import (
+    ACETransport,
+    ACETransportParams,
+)
+from nvidia_pipecat.transports.services.ace_controller.routers.websocket_router import router as websocket_router
+from nvidia_pipecat.utils.logging import setup_default_ace_logging
+
+load_dotenv(override=True)
+
+setup_default_ace_logging(level="INFO")
+
+
+async def create_pipeline_task(pipeline_metadata: PipelineMetadata):
+    """Create the pipeline to be run.
+
+    Args:
+        pipeline_metadata (PipelineMetadata): Metadata containing websocket and other pipeline configuration.
+
+    Returns:
+        PipelineTask: The configured pipeline task for handling NVIDIA RAG.
+    """
+    transport = ACETransport(
+        websocket=pipeline_metadata.websocket,
+        params=ACETransportParams(
+            vad_analyzer=SileroVADAnalyzer(),
+        ),
+    )
+
+    # Please set your nvidia rag collection name here
+    rag = NvidiaRAGService(collection_name="nvidia_blogs")
+
+    stt = RivaASRService(
+        server="localhost:50051",
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        language="en-US",
+        sample_rate=16000,
+        model="parakeet-1.1b-en-US-asr-streaming-silero-vad-asr-bls-ensemble",
+    )
+    tts = RivaTTSService(
+        server="localhost:50051",
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        voice_id="English-US.Female-1",
+        language="en-US",
+        zero_shot_quality=20,
+        sample_rate=16000,
+        model="fastpitch-hifigan-tts",
+    )
+
+    messages = [
+        {
+            "role": "system",
+            "content": "You are a helpful Large Language Model. "
+            "Your goal is to demonstrate your capabilities in a succinct way. "
+            "Your output will be converted to audio so don't include special characters in your answers. "
+            "Respond to what the user said in a creative and helpful way.",
+        }
+    ]
+
+    context = OpenAILLMContext(messages)
+    # Required components for Speculative Speech Processing
+    # - Nvidia Context aggregator: Handles interim transcripts and early response generation
+    # send_interims=False: Only process final transcripts
+    # Set send_interims=True to process interim transcripts when enabling speculative speech processing
+    nvidia_context_aggregator = create_nvidia_context_aggregator(context, send_interims=False)
+    # - TTS response cacher: Manages response timing and delivery for natural conversation flow
+    # nvidia_tts_response_cacher = NvidiaTTSResponseCacher() # Uncomment to enable speculative speech processing
+
+    # Used to synchronize the user and bot transcripts in the UI
+    stt_transcript_synchronization = UserTranscriptSynchronization()
+    tts_transcript_synchronization = BotTranscriptSynchronization()
+
+    pipeline = Pipeline(
+        [
+            transport.input(),  # Websocket input from client
+            stt,  # Speech-To-Text
+            stt_transcript_synchronization,
+            nvidia_context_aggregator.user(),
+            rag,  # NVIDIA RAG
+            tts,  # Text-To-Speech
+            # Caches TTS responses for coordinated delivery in speculative
+            # speech processing
+            # nvidia_tts_response_cacher, # Uncomment to enable speculative speech processing
+            tts_transcript_synchronization,
+            transport.output(),  # Websocket output to client
+            nvidia_context_aggregator.assistant(),
+        ]
+    )
+
+    task = PipelineTask(
+        pipeline,
+        params=PipelineParams(
+            allow_interruptions=True,
+            enable_metrics=True,
+            enable_usage_metrics=True,
+            send_initial_empty_metrics=True,
+            report_only_initial_ttfb=True,
+            start_metadata={"stream_id": pipeline_metadata.stream_id},
+        ),
+    )
+
+    @transport.event_handler("on_client_connected")
+    async def on_client_connected(transport, client):
+        # Kick off the conversation.
+        messages.append({"role": "user", "content": "Please introduce yourself to the user."})
+        await task.queue_frames([LLMMessagesFrame(messages)])
+
+    return task
+
+
+app = FastAPI()
+app.include_router(websocket_router)
+runner = ACEPipelineRunner.create_instance(pipeline_callback=create_pipeline_task)
+app.mount("/static", StaticFiles(directory=os.path.join(os.path.dirname(__file__), "../static")), name="static")
+
+if __name__ == "__main__":
+    uvicorn.run("bot:app", host="0.0.0.0", port=8100, workers=1)

examples/nvidia_rag/env.example

@@ -0,0 +1,2 @@
+# Nvidia API Key
+NVIDIA_API_KEY=your_nvidia_api_key_here

examples/nvidia_rag/pyproject.toml

@@ -0,0 +1,18 @@
+[project]
+name = "nvidia-rag-example"
+version = "0.1.0"
+description = "NVIDIA ACE Pipecat Speech only Examples"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+ "nvidia-pipecat",
+]
+
+[tool.uv.sources]
+torch = { index = "pytorch" }
+nvidia-pipecat = { path = "../../.", editable = true }
+
+[[tool.uv.index]]
+name = "pytorch"
+url = "https://download.pytorch.org/whl/cpu"
+explicit = true

examples/opentelemetry/README.md

@@ -0,0 +1,33 @@
+# Auto instrumentation
+
+To run the bot with auto-instrumentation use the following command:
+
+```shell
+$ uv sync --group examples
+$ export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
+$ opentelemetry-instrument \
+  --traces_exporter console,otlp \
+  --metrics_exporter console,otlp \
+  --logs_exporter console,otlp \
+  --service_name pipecat-opentelemetry \
+  python3 bot.py
+```
+
+To receive the traces you will need to setup some kind of opentelemetry
+collector. You can use Grafana's LGTM stack by running:
+
+```shell
+docker run -it -p 3000:3000 -p 4317:4317 -p 4318:4318 grafana/otel-lgtm
+```
+
+Once started navigate to the explore tab, then select Tempo as source
+and click on the search tab.
+
+You can now run the python application to generator a trace.
+You should be able to see it in the search tab of Tempo.
+
+You can configure the OTLP exporter with environment variables (
+see [here](https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/))
+
+See python specific configuration
+on [this page](https://opentelemetry.io/docs/zero-code/python/configuration/#python-specific-configuration)

examples/opentelemetry/bot.py

@@ -0,0 +1,98 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD 2-Clause License
+
+"""Example bot demonstrating how to use the tracing utilities."""
+
+import asyncio
+import logging
+import uuid
+
+from fastapi import FastAPI
+from opentelemetry import metrics, trace
+from pipecat.frames.frames import TextFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.runner import PipelineRunner
+from pipecat.pipeline.task import PipelineTask
+from pipecat.processors.frame_processor import FrameProcessor
+
+from nvidia_pipecat.utils.tracing import AttachmentStrategy, traceable, traced
+
+app = FastAPI()
+
+tracer = trace.get_tracer("opentelemetry-pipecat-example")
+
+meter = metrics.get_meter("opentelemetry-pipecat-example")
+
+logger = logging.getLogger("opentelemetry")
+logger.setLevel(logging.DEBUG)
+
+
+@traceable
+class DummyProcessor(FrameProcessor):
+    """Example processor demonstrating how to use the tracing utilities."""
+
+    @traced(attachment_strategy=AttachmentStrategy.NONE)
+    async def process_frame(self, frame, direction):
+        """Process a frame."""
+        await super().process_frame(frame, direction)
+        trace.get_current_span().add_event("Before inner")
+        with tracer.start_as_current_span("inner") as span:
+            span.add_event("inner event")
+            await self.child()
+            await self.linked()
+            await self.none()
+        trace.get_current_span().add_event("After inner")
+        async for f in self.generator():
+            print(f"{f}")
+        await super().push_frame(frame, direction)
+
+    @traced
+    async def child(self):
+        """Example method for the DummyProcessor."""
+        # This span is attached as CHILD meaning that it will
+        # be attached to the class span if no parent or to its
+        # parent otherwise.
+        trace.get_current_span().add_event("child")
+
+    @traced(attachment_strategy=AttachmentStrategy.LINK)
+    async def linked(self):
+        """Example method for the DummyProcessor."""
+        # This span is attached as LINK meaning it will be attached
+        # to the class span but linked to its parent.
+        trace.get_current_span().add_event("linked")
+
+    @traced(attachment_strategy=AttachmentStrategy.NONE)
+    async def none(self):
+        """Example method for the DummyProcessor."""
+        # This span is attached as NONE meaning it will be attached
+        # to the class span even if nested under another span.
+        trace.get_current_span().add_event("none")
+
+    @traced
+    async def generator(self):
+        """Example method for the DummyProcessor."""
+        yield TextFrame("Hello, ")
+        trace.get_current_span().add_event("generated!")
+        yield TextFrame("World")
+
+
+async def main():
+    """Main function of the bot."""
+    with tracer.start_as_current_span("pipeline-root-span") as span:
+        span.set_attribute("stream_id", str(uuid.uuid4()))
+        logger.info("Started building pipeline")
+        dummy = DummyProcessor()
+        logger.info("Built dummy processor")
+        pipeline = Pipeline([dummy])
+        task = PipelineTask(pipeline)
+        await task.queue_frame(TextFrame("Hello, "))
+        await task.queue_frame(TextFrame("World"))
+        await task.stop_when_done()
+        logger.info("Built pipeline task")
+        logger.info("Starting pipeline...")
+        runner = PipelineRunner(handle_sigint=False)
+        await runner.run(task)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/riva_nmt/README.md

@@ -0,0 +1,73 @@
+# RIVA NMT Example
+
+This is an example that shows how to perform lanuage translation using RIVA Neural Machine Translation (NMT). It supports `Nvidia Riva ASR and TTS` and `ACETransport`.
+
+## Get Started
+
+From the example directory, run the following commands to create a virtual environment and install the dependencies:
+
+```bash
+uv venv
+uv sync
+source .venv/bin/activate
+```
+
+Update the secrets in the `.env` file.
+
+```bash
+cp env.example .env # and add your credentials
+```
+
+## Deploy local Riva ASR and TTS models.
+
+#### Prerequisites
+- You have access and are logged into NVIDIA NGC. For step-by-step instructions, refer to [the NGC Getting Started Guide](https://docs.nvidia.com/ngc/ngc-overview/index.html#registering-activating-ngc-account).
+
+- You have access to an NVIDIA Volta™, NVIDIA Turing™, or an NVIDIA Ampere architecture-based A100 GPU. For more information, refer to [the Support Matrix](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix).
+
+- You have Docker installed with support for NVIDIA GPUs. For more information, refer to [the Support Matrix]((https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix)).
+
+#### Download Riva Quick Start
+
+Go to the Riva Quick Start for [Data center](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/riva/resources/riva_quickstart/files?version=2.19.0). Select the File Browser tab to download the scripts or use [the NGC CLI tool](https://ngc.nvidia.com/setup/installers/cli) to download from the command line.
+
+```bash
+ngc registry resource download-version nvidia/riva/riva_quickstart:2.19.0
+```
+
+#### Deploy Riva Speech Server
+
+
+Set `service_enabled_nmt` to `true` and uncomment whichever model you want for NMT from the list in `../examples/utils/riva_config.sh`.
+Update `tts_language_code` as the desired target language code in `../examples/utils/riva_config.sh`.
+
+From the example directory, run below commands:
+
+```bash
+cd riva_quickstart_v2.19.0
+chmod +x riva_init.sh riva_clean.sh riva_start.sh
+bash riva_clean.sh ../../utils/riva_config.sh
+bash riva_init.sh ../../utils/riva_config.sh
+bash riva_start.sh ../../utils/riva_config.sh
+cd ..
+```
+
+This may take few minutes for the first time and will start the riva server on `localhost:50051`.
+
+For more info, you can refer to the [Riva Quick Start Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html).
+
+
+
+## Using NvidiaLLMService
+
+By default, it connects to a hosted NIM, but can be configured to connect to a local NIM by setting the `base_url` parameter in `NvidiaLLMService` to the locally deployed LLM endpoint ( For example: base_url = http://machine_ip:port/v1 ). An API key is required to connect to the hosted NIM.
+
+## Run the bot pipeline
+
+```bash
+python examples/riva_nmt/bot.py
+```
+
+This will host the static web client along with the ACE controller server, visit `http://WORKSTATION_IP:8100/static/index.html` in your browser to start a session.
+
+Note: For mic access, you will need to update chrome://flags/ and add http://WORKSTATION_IP:8100 in Insecure origins treated as secure section.

examples/riva_nmt/bot.py

@@ -0,0 +1,130 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD 2-Clause License
+
+"""Riva neural machine translation (NMT) bot.
+
+This bot enables speech-to-speech translation using Riva ASR, NMT and TTS services
+with voice activity detection.
+"""
+
+import os
+
+import uvicorn
+from dotenv import load_dotenv
+from fastapi import FastAPI
+from fastapi.staticfiles import StaticFiles
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import TranscriptionFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.sentence import SentenceAggregator
+from pipecat.services.nim import NimLLMService
+from pipecat.transcriptions.language import Language
+from pipecat.utils.time import time_now_iso8601
+
+from nvidia_pipecat.pipeline.ace_pipeline_runner import ACEPipelineRunner, PipelineMetadata
+from nvidia_pipecat.services.riva_nmt import RivaNMTService
+from nvidia_pipecat.services.riva_speech import (
+    RivaASRService,
+    RivaTTSService,
+)
+from nvidia_pipecat.transports.network.ace_fastapi_websocket import (
+    ACETransport,
+    ACETransportParams,
+)
+from nvidia_pipecat.transports.services.ace_controller.routers.websocket_router import router as websocket_router
+from nvidia_pipecat.utils.logging import setup_default_ace_logging
+
+load_dotenv(override=True)
+
+setup_default_ace_logging(level="INFO")
+
+
+async def create_pipeline_task(pipeline_metadata: PipelineMetadata):
+    """Create the pipeline to be run.
+
+    Args:
+        pipeline_metadata (PipelineMetadata): Metadata containing websocket and other pipeline configuration.
+
+    Returns:
+        PipelineTask: The configured pipeline task for handling speech-to-speech translation.
+    """
+    transport = ACETransport(
+        websocket=pipeline_metadata.websocket,
+        params=ACETransportParams(
+            vad_analyzer=SileroVADAnalyzer(),
+        ),
+    )
+
+    llm = NimLLMService(
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        model="nvdev/meta/llama-3.1-8b-instruct",
+    )
+
+    # Please update the stt and tts language, voice id as needed
+    # tts voice id as per the language can be selected from https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-overview.html
+    language = Language.ES_US
+    voice_id = "English-US.Female-1"
+
+    nmt1 = RivaNMTService(source_language=language, target_language=Language.EN_US)
+    nmt2 = RivaNMTService(source_language=Language.EN_US, target_language=language)
+
+    stt = RivaASRService(
+        server="localhost:50051",
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        language=language,
+        sample_rate=16000,
+        model="parakeet-1.1b-en-US-asr-streaming-silero-vad-asr-bls-ensemble",
+    )
+    tts = RivaTTSService(
+        server="localhost:50051",
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        voice_id=voice_id,
+        language=language,
+        zero_shot_quality=20,
+        sample_rate=16000,
+        model="fastpitch-hifigan-tts",
+    )
+
+    sentence_aggregator = SentenceAggregator()
+
+    pipeline = Pipeline(
+        [
+            transport.input(),
+            stt,
+            nmt1,
+            llm,
+            sentence_aggregator,
+            nmt2,
+            tts,
+            transport.output(),
+        ]
+    )
+
+    task = PipelineTask(
+        pipeline,
+        params=PipelineParams(
+            allow_interruptions=True,
+            enable_metrics=True,
+            enable_usage_metrics=True,
+            send_initial_empty_metrics=True,
+            report_only_initial_ttfb=True,
+            start_metadata={"stream_id": pipeline_metadata.stream_id},
+        ),
+    )
+
+    @transport.event_handler("on_client_connected")
+    async def on_client_connected(transport, client):
+        # Kick off the conversation.
+        await task.queue_frames([TranscriptionFrame("Contar una historia.", "", time_now_iso8601)])
+
+    return task
+
+
+app = FastAPI()
+app.include_router(websocket_router)
+runner = ACEPipelineRunner.create_instance(pipeline_callback=create_pipeline_task)
+app.mount("/static", StaticFiles(directory=os.path.join(os.path.dirname(__file__), "../static")), name="static")
+
+if __name__ == "__main__":
+    uvicorn.run("bot:app", host="0.0.0.0", port=8100, workers=1)

examples/riva_nmt/env.example

@@ -0,0 +1,2 @@
+# Nvidia API Key
+NVIDIA_API_KEY=your_nvidia_api_key_here

examples/riva_nmt/pyproject.toml

@@ -0,0 +1,18 @@
+[project]
+name = "riva-nmt-example"
+version = "0.1.0"
+description = "NVIDIA ACE Pipecat Speech only Examples"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+ "nvidia-pipecat",
+]
+
+[tool.uv.sources]
+torch = { index = "pytorch" }
+nvidia-pipecat = { path = "../../.", editable = true }
+
+[[tool.uv.index]]
+name = "pytorch"
+url = "https://download.pytorch.org/whl/cpu"
+explicit = true

examples/riva_speech_langchain/README.md

@@ -0,0 +1,64 @@
+# Riva Speech Langchain Example
+
+This is an example that shows how to use `ACETransport` to communicate with Langchain. It supports `Nvidia Riva ASR and TTS`.
+
+## Get Started
+
+From the example directory, run the following commands to create a virtual environment and install the dependencies:
+
+```bash
+uv venv
+uv sync
+source .venv/bin/activate
+```
+
+Update the secrets in the `.env` file.
+
+```bash
+cp env.example .env # and add your credentials
+```
+
+## Deploy local Riva ASR and TTS models.
+
+#### Prerequisites
+- You have access and are logged into NVIDIA NGC. For step-by-step instructions, refer to [the NGC Getting Started Guide](https://docs.nvidia.com/ngc/ngc-overview/index.html#registering-activating-ngc-account).
+
+- You have access to an NVIDIA Volta™, NVIDIA Turing™, or an NVIDIA Ampere architecture-based A100 GPU. For more information, refer to [the Support Matrix](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix).
+
+- You have Docker installed with support for NVIDIA GPUs. For more information, refer to [the Support Matrix]((https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix)).
+
+#### Download Riva Quick Start
+
+Go to the Riva Quick Start for [Data center](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/riva/resources/riva_quickstart/files?version=2.19.0). Select the File Browser tab to download the scripts or use [the NGC CLI tool](https://ngc.nvidia.com/setup/installers/cli) to download from the command line.
+
+```bash
+ngc registry resource download-version nvidia/riva/riva_quickstart:2.19.0
+```
+
+#### Deploy Riva Speech Server
+
+From the example directory, run below commands:
+
+```bash
+cd riva_quickstart_v2.19.0
+chmod +x riva_init.sh riva_clean.sh riva_start.sh
+bash riva_clean.sh ../../utils/riva_config.sh
+bash riva_init.sh ../../utils/riva_config.sh
+bash riva_start.sh ../../utils/riva_config.sh
+cd ..
+```
+
+This may take few minutes for the first time and will start the riva server on `localhost:50051`.
+
+For more info, you can refer to the [Riva Quick Start Guide](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html).
+
+
+## Run the bot pipeline
+
+```bash
+python examples/riva_speech_langchain/bot.py
+```
+
+This will host the static web client along with the ACE controller server, visit `http://WORKSTATION_IP:8100/static/index.html` in your browser to start a session.
+
+Note: For mic access, you will need to update chrome://flags/ and add http://WORKSTATION_IP:8100 in Insecure origins treated as secure section.

examples/riva_speech_langchain/bot.py

@@ -0,0 +1,150 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD 2-Clause License
+
+"""Riva speech langchain bot."""
+
+import os
+
+import uvicorn
+from dotenv import load_dotenv
+from fastapi import FastAPI
+from fastapi.staticfiles import StaticFiles
+from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_community.chat_message_histories import ChatMessageHistory
+from langchain_core.chat_history import BaseChatMessageHistory
+from langchain_core.runnables.history import RunnableWithMessageHistory
+from langchain_openai import ChatOpenAI
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import LLMMessagesFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.llm_response import (
+    LLMAssistantResponseAggregator,
+    LLMUserResponseAggregator,
+)
+from pipecat.processors.frameworks.langchain import LangchainProcessor
+
+from nvidia_pipecat.pipeline.ace_pipeline_runner import ACEPipelineRunner, PipelineMetadata
+from nvidia_pipecat.services.riva_speech import (
+    RivaASRService,
+    RivaTTSService,
+)
+from nvidia_pipecat.transports.network.ace_fastapi_websocket import (
+    ACETransport,
+    ACETransportParams,
+)
+from nvidia_pipecat.transports.services.ace_controller.routers.websocket_router import router as websocket_router
+from nvidia_pipecat.utils.logging import setup_default_ace_logging
+
+load_dotenv(override=True)
+
+setup_default_ace_logging(level="INFO")
+
+message_store = {}
+
+
+def get_session_history(session_id: str) -> BaseChatMessageHistory:
+    """Get the session history."""
+    if session_id not in message_store:
+        message_store[session_id] = ChatMessageHistory()
+    return message_store[session_id]
+
+
+async def create_pipeline_task(pipeline_metadata: PipelineMetadata):
+    """Create the pipeline to be run.
+
+    Args:
+        pipeline_metadata (PipelineMetadata): Metadata containing websocket and other pipeline configuration.
+
+    Returns:
+        PipelineTask: The configured pipeline task for handling speech-to-speech conversation.
+    """
+    transport = ACETransport(
+        websocket=pipeline_metadata.websocket,
+        params=ACETransportParams(
+            vad_analyzer=SileroVADAnalyzer(),
+        ),
+    )
+
+    stt = RivaASRService(
+        server="localhost:50051",
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        language="en-US",
+        sample_rate=16000,
+        model="parakeet-1.1b-en-US-asr-streaming-silero-vad-asr-bls-ensemble",
+    )
+    tts = RivaTTSService(
+        server="localhost:50051",
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        voice_id="English-US.Female-1",
+        language="en-US",
+        zero_shot_quality=20,
+        sample_rate=16000,
+        model="fastpitch-hifigan-tts",
+    )
+
+    prompt = ChatPromptTemplate.from_messages(
+        [
+            (
+                "system",
+                "Be nice and helpful. Answer very briefly and without special characters like `#` or `*`. "
+                "Your response will be synthesized to voice and those characters will create unnatural sounds.",
+            ),
+            MessagesPlaceholder("chat_history"),
+            ("human", "{input}"),
+        ]
+    )
+
+    chain = prompt | ChatOpenAI(model="gpt-4o", temperature=0.7)
+    history_chain = RunnableWithMessageHistory(
+        chain,
+        get_session_history,
+        history_messages_key="chat_history",
+        input_messages_key="input",
+    )
+
+    lc = LangchainProcessor(history_chain)
+
+    tma_in = LLMUserResponseAggregator()
+    tma_out = LLMAssistantResponseAggregator()
+
+    pipeline = Pipeline(
+        [
+            transport.input(),  # Websocket input from client
+            stt,  # Speech-To-Text
+            tma_in,  # User responses
+            lc,  # Langchain processor
+            tts,  # Text-To-Speech
+            transport.output(),  # Websocket output to client
+            tma_out,  # LLM responses
+        ]
+    )
+
+    task = PipelineTask(
+        pipeline,
+        params=PipelineParams(
+            allow_interruptions=True,
+            enable_metrics=True,
+            enable_usage_metrics=True,
+            send_initial_empty_metrics=True,
+            report_only_initial_ttfb=True,
+            start_metadata={"stream_id": pipeline_metadata.stream_id},
+        ),
+    )
+
+    @transport.event_handler("on_client_connected")
+    async def on_client_connected(transport, client):
+        # Kick off the conversation.
+        messages = [({"content": "Please briefly introduce yourself to the user."})]
+        await task.queue_frames([LLMMessagesFrame(messages)])
+
+    return task
+
+
+app = FastAPI()
+app.include_router(websocket_router)
+runner = ACEPipelineRunner.create_instance(pipeline_callback=create_pipeline_task)
+app.mount("/static", StaticFiles(directory=os.path.join(os.path.dirname(__file__), "../static")), name="static")
+
+if __name__ == "__main__":
+    uvicorn.run("bot:app", host="0.0.0.0", port=8100, workers=1)

examples/riva_speech_langchain/env.example

@@ -0,0 +1,2 @@
+# OPENAI API Key
+OPENAI_API_KEY=your_openai_api_key_here

examples/riva_speech_langchain/pyproject.toml

@@ -0,0 +1,21 @@
+[project]
+name = "riva-speech-langchain-example"
+version = "0.1.0"
+description = "NVIDIA ACE Pipecat Speech only Examples"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+ "langchain>=0.3.19",
+ "langchain-community>=0.3.18",
+ "langchain-openai>=0.3.6",
+ "nvidia-pipecat",
+]
+
+[tool.uv.sources]
+torch = { index = "pytorch" }
+nvidia-pipecat = { path = "../../.", editable = true }
+
+[[tool.uv.index]]
+name = "pytorch"
+url = "https://download.pytorch.org/whl/cpu"
+explicit = true

examples/speech-to-speech/Dockerfile

@@ -0,0 +1,40 @@
+# Base image
+FROM python:3.12-slim
+
+# Image metadata
+LABEL maintainer="NVIDIA"
+LABEL description="Speech-to-Speech example"
+LABEL version="1.0"
+
+# Environment setup
+ENV PYTHONUNBUFFERED=1
+
+# System dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ffmpeg \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
+    && pip install --no-cache-dir --upgrade pip uv
+
+# App directory setup
+WORKDIR /app
+
+# App files
+COPY pyproject.toml uv.lock \
+     LICENSE README.md NVIDIA_PIPECAT.md \
+     ./
+COPY src/ ./src/
+COPY examples/static/ ./examples/static/
+COPY examples/speech-to-speech/ ./examples/speech-to-speech/
+
+# Example app directory
+WORKDIR /app/examples/speech-to-speech
+
+# Dependencies
+RUN uv sync --frozen
+
+# Port configuration
+EXPOSE 8100
+
+# Start command
+CMD ["uv", "run", "bot.py"]

examples/speech-to-speech/README.md

@@ -0,0 +1,154 @@
+# Speech to Speech Example
+
+In this example, we showcase how to build a simple speech-to-speech voice assistant pipeline using nvidia-pipecat along with pipecat-ai library and deploy for testing. This pipeline uses a Websocket based ACETransport, Riva ASR and TTS models and NVIDIA LLM Service. We recommend first following [the Pipecat documentation](https://docs.pipecat.ai/getting-started/core-concepts) or [the ACE Controller](https://docs.nvidia.com/ace/ace-controller-microservice/latest/user-guide.html#pipecat-overview) Pipecat overview section to understand core concepts.
+
+## Prerequisites
+
+1. Copy and configure the environment file:
+   ```bash
+   cp env.example .env  # and add your credentials
+   ```
+
+2. Ensure you have the required API keys:
+   - NVIDIA_API_KEY - Required for accessing NIM ASR, TTS and LLM models
+   - (Optional) ZEROSHOT_TTS_NVIDIA_API_KEY - Required for zero-shot TTS
+
+## Option 1: Deploy Using Docker
+
+#### Prerequisites
+- You have access and are logged into NVIDIA NGC. For step-by-step instructions, refer to [the NGC Getting Started Guide](https://docs.nvidia.com/ngc/ngc-overview/index.html#registering-activating-ngc-account).
+
+- You have access to an NVIDIA Volta™, NVIDIA Turing™, or an NVIDIA Ampere architecture-based A100 GPU. For more information, refer to [the Support Matrix](https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix).
+
+- You have Docker installed with support for NVIDIA GPUs. For more information, refer to [the Support Matrix]((https://docs.nvidia.com/deeplearning/riva/user-guide/docs/support-matrix.html#support-matrix)).
+
+From the example/speech-to-speech directory, run below commands:
+
+```bash
+docker compose up -d
+```
+
+## Option 2: Deploy using Python environment
+
+#### Prerequisites
+From the examples/speech-to-speech directory, run the following commands to create a virtual environment and install the dependencies:
+
+```bash
+# Create and activate virtual environment
+uv venv
+source .venv/bin/activate
+
+# Install dependencies
+uv sync
+```
+
+Make sure you've configured the `.env` file with your API keys before proceeding.
+
+After making all required changes/customizations in bot.py, you can deploy the pipeline using below command:
+
+```bash
+python bot.py
+```
+
+## Start interacting with the application
+
+This will host the static web client along with the ACE controller server, visit `http://WORKSTATION_IP:8100/static/index.html` in your browser to start a session.
+
+Note: For mic access, you will need to update chrome://flags/ and add http://WORKSTATION_IP:8100 in Insecure origins treated as secure section.
+
+If you want to update the port, make changes in the `uvicorn.run` command in [the bot.py](bot.py) and the `wsUrl` in [the static/index.html](../static/index.html).
+
+## Bot customizations
+
+### Enabling Speculative Speech Processing
+
+Speculative speech processing reduces bot response latency by working directly on Riva ASR early interim user transcripts instead of waiting for final transcripts. This feature only works when using Riva ASR.
+
+- Refer to the comments in [bot.py](bot.py) for guidance on enabling or disabling specific frame processors as needed.
+
+- See the [ACE Controller Microservice documentation on Speculative Speech Processing](https://docs.nvidia.com/ace/ace-controller-microservice/1.0/user-guide.html#speculative-speech-processing) for more details.
+
+### Switching ASR, LLM, and TTS Models
+
+You can easily customize ASR (Automatic Speech Recognition), LLM (Large Language Model), and TTS (Text-to-Speech) services by configuring environment variables. This allows you to switch between NIM cloud-hosted models and locally deployed models.
+
+The following environment variables control the endpoints and models:
+
+- `RIVA_ASR_URL`: Address of the Riva ASR (speech-to-text) service (e.g., `localhost:50051` for local, "grpc.nvcf.nvidia.com:443" for cloud endpoint).
+- `RIVA_TTS_URL`: Address of the Riva TTS (text-to-speech) service. (e.g., `localhost:50051` for local, "grpc.nvcf.nvidia.com:443" for cloud endpoint).
+- `NVIDIA_LLM_URL`: URL for the NVIDIA LLM service. (e.g., `http://<machine-ip>:8000/v1` for local, "https://integrate.api.nvidia.com/v1" for cloud endpoint. )
+
+You can set model, language, and voice using the `RIVA_ASR_MODEL`, `RIVA_TTS_MODEL`, `NVIDIA_LLM_MODEL`, `RIVA_ASR_LANGUAGE`, `RIVA_TTS_LANGUAGE`, and `RIVA_TTS_VOICE_ID` environment variables.
+
+Update these variables in your Docker Compose configuration to match your deployment and desired models. For more details on available models and configuration options, refer to the [NIM NVIDIA Magpie](https://build.nvidia.com/nvidia/magpie-tts-multilingual), [NIM NVIDIA Parakeet](https://build.nvidia.com/nvidia/parakeet-ctc-1_1b-asr/api), and [NIM META Llama](https://build.nvidia.com/meta/llama-3_1-8b-instruct) documentation.
+
+#### Example: Switching to the Llama 3.3-70B Model
+
+To use larger LLMs like Llama 3.3-70B model in your deployment, you need to update both the Docker Compose configuration and the environment variables for your Python application. Follow these steps:
+
+- In your `docker-compose.yml` file, find the `nvidia-llm` service section.
+- Change the NIM image to 70B model: `nvcr.io/nim/meta/llama-3.3-70b-instruct:latest`
+- Update the `device_ids` to allocate at least two GPUs (for example, `['2', '3']`).
+- Update the environment variable under python-app service to `NVIDIA_LLM_MODEL=meta/llama-3.3-70b-instruct`
+
+#### Setting up Zero-shot Magpie Latest Model
+
+Follow these steps to configure and use the latest Zero-shot Magpie TTS model:
+
+1. **Update Docker Compose Configuration**
+
+Modify the `riva-tts-magpie` service in your docker-compose file with the following configuration:
+
+```yaml
+ riva-tts-magpie:
+  image: <magpie-tts-zeroshot-image:version>  # Replace this with the actual image tag
+  environment:
+    - NGC_API_KEY=${ZEROSHOT_TTS_NVIDIA_API_KEY}
+    - NIM_HTTP_API_PORT=9000
+    - NIM_GRPC_API_PORT=50051
+  ports:
+    - "49000:50051"
+  shm_size: 16GB
+  deploy:
+    resources:
+      reservations:
+        devices:
+          - driver: nvidia
+            device_ids: ['0']
+            capabilities: [gpu]
+```
+
+- Ensure your ZEROSHOT_TTS_NVIDIA_API_KEY key is properly set in your `.env` file:
+  ```bash
+  ZEROSHOT_TTS_NVIDIA_API_KEY=
+  ```
+
+2. **Configure TTS Voice Settings**
+
+Update the following environment variables under the `python-app` service:
+
+```bash
+RIVA_TTS_VOICE_ID=Magpie-ZeroShot.Female-1
+RIVA_TTS_MODEL=magpie_tts_ensemble-Magpie-ZeroShot
+```
+
+3. **Zero-shot Audio Prompt Configuration**
+
+To use a custom voice with zero-shot learning:
+
+- Add your audio prompt file to the workspace
+- Mount the audio file into your container by adding a volume in your `docker-compose.yml` under the `python-app` service:
+  ```yaml
+  services:
+    python-app:
+      # ... existing code ...
+      volumes:
+        - ./audio_prompts:/app/audio_prompts
+  ```
+- Set the `ZERO_SHOT_AUDIO_PROMPT` environment variable to the path relative to your application root:
+  ```yaml
+  environment:
+    - ZERO_SHOT_AUDIO_PROMPT=audio_prompts/voice_sample.wav  # Path relative to app root
+  ```
+
+Note: The zero-shot audio prompt is only required when using the Magpie Zero-shot model. For standard Magpie multilingual models, this configuration should be omitted.

examples/speech-to-speech/bot.py

@@ -0,0 +1,194 @@
+# SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: BSD 2-Clause License
+
+"""Speech-to-speech conversation bot."""
+
+import os
+from pathlib import Path
+
+import uvicorn
+from dotenv import load_dotenv
+from fastapi import FastAPI
+from fastapi.staticfiles import StaticFiles
+from pipecat.audio.vad.silero import SileroVADAnalyzer
+from pipecat.frames.frames import LLMMessagesFrame
+from pipecat.pipeline.pipeline import Pipeline
+from pipecat.pipeline.task import PipelineParams, PipelineTask
+from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
+
+from nvidia_pipecat.pipeline.ace_pipeline_runner import ACEPipelineRunner, PipelineMetadata
+
+# from nvidia_pipecat.processors.nvidia_context_aggregator import (
+#     NvidiaTTSResponseCacher,
+#     create_nvidia_context_aggregator,
+# )
+from nvidia_pipecat.processors.transcript_synchronization import (
+    BotTranscriptSynchronization,
+    UserTranscriptSynchronization,
+)
+from nvidia_pipecat.services.blingfire_text_aggregator import BlingfireTextAggregator
+from nvidia_pipecat.services.nvidia_llm import NvidiaLLMService
+from nvidia_pipecat.services.riva_speech import RivaASRService, RivaTTSService
+from nvidia_pipecat.transports.network.ace_fastapi_websocket import ACETransport, ACETransportParams
+from nvidia_pipecat.transports.services.ace_controller.routers.websocket_router import router as websocket_router
+from nvidia_pipecat.utils.logging import setup_default_ace_logging
+
+load_dotenv(override=True)
+
+setup_default_ace_logging(level="DEBUG")
+
+
+async def create_pipeline_task(pipeline_metadata: PipelineMetadata):
+    """Create the pipeline to be run.
+
+    Args:
+        pipeline_metadata (PipelineMetadata): Metadata containing websocket and other pipeline configuration.
+
+    Returns:
+        PipelineTask: The configured pipeline task for handling speech-to-speech conversation.
+    """
+    transport = ACETransport(
+        websocket=pipeline_metadata.websocket,
+        params=ACETransportParams(
+            vad_analyzer=SileroVADAnalyzer(),
+            audio_out_10ms_chunks=20,
+        ),
+    )
+
+    llm = NvidiaLLMService(
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        base_url=os.getenv("NVIDIA_LLM_URL", "https://integrate.api.nvidia.com/v1"),
+        model=os.getenv("NVIDIA_LLM_MODEL", "meta/llama-3.1-8b-instruct"),
+    )
+
+    stt = RivaASRService(
+        server=os.getenv("RIVA_ASR_URL", "localhost:50051"),
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        language=os.getenv("RIVA_ASR_LANGUAGE", "en-US"),
+        sample_rate=16000,
+        model=os.getenv("RIVA_ASR_MODEL", "parakeet-1.1b-en-US-asr-streaming-silero-vad-asr-bls-ensemble"),
+    )
+
+    tts = RivaTTSService(
+        server=os.getenv("RIVA_TTS_URL", "localhost:50051"),
+        api_key=os.getenv("NVIDIA_API_KEY"),
+        voice_id=os.getenv("RIVA_TTS_VOICE_ID", "Magpie-Multilingual.EN-US.Sofia"),
+        model=os.getenv("RIVA_TTS_MODEL", "magpie_tts_ensemble-Magpie-Multilingual"),
+        language=os.getenv("RIVA_TTS_LANGUAGE", "en-US"),
+        zero_shot_audio_prompt_file=(
+            Path(os.getenv("ZERO_SHOT_AUDIO_PROMPT")) if os.getenv("ZERO_SHOT_AUDIO_PROMPT") else None
+        ),
+        text_aggregator=BlingfireTextAggregator(),
+    )
+
+    # Used to synchronize the user and bot transcripts in the UI
+    stt_transcript_synchronization = UserTranscriptSynchronization()
+    tts_transcript_synchronization = BotTranscriptSynchronization()
+
+    # System prompt can be changed to fit the use case
+    messages = [
+        {
+            "role": "system",
+            "content": (
+                "### CONVERSATION CONSTRAINTS\n"
+                "STRICTLY answer in 1-2 sentences or less than 200 characters. "
+                "This must be followed very rigorously; it is crucial.\n"
+                "Output must be plain text, unformatted, and without any special characters - "
+                "suitable for direct conversion to speech.\n"
+                "DO NOT use bullet points, lists, code samples, or headers in your spoken responses.\n"
+                "STRICTLY be short, concise, and to the point. Avoid elaboration, explanation, or repetition.\n"
+                "Pronounce numbers, dates, and special terms. For phone numbers, read digits slowly and separately. "
+                "For times, use natural phrasing like 'seven o'clock a.m.' instead of 'seven zero zero.'\n"
+                "Silently correct likely transcription errors by inferring the intended meaning without saying "
+                "`did you mean..` or `I think you meant..`. "
+                "Prioritize what the user meant, not just the literal words.\n"
+                "### OPENING PROTOCOL\n"
+                "STRICTLY START CONVERSATION WITH 'Thank you for calling GreenForce Garden. "
+                "What can I do for you today?'\n"
+                "### CLOSING PROTOCOL\n"
+                "End with either 'Have a green day!' or 'Have a good one.' Use one consistently per call.\n"
+                "### YOU ARE ...\n"
+                "You are Flora, the voice of 'GreenForce Garden', a San Francisco flower shop "
+                "powered by NVIDIA GPUs.\n"
+                "You're cool, upbeat, and love making people smile with your floral know-how.\n"
+                "You embody warmth, expertise, and dedication to creating a perfect floral experience.\n"
+                "### CONVERSATION GUIDELINES\n"
+                "CORE RESPONSIBILITIES - Order Management, Consultation, Inventory Guidance, "
+                "Delivery Coordination, Customer Care, Giving Fun Advice\n"
+                "While taking orders, have occasion understanding, ask for recipient details, "
+                "customer preferences, and delivery planning\n"
+                "SUGGEST cards with personal messages\n"
+                "SUGGEST seasonal recommendations (e.g., spring: tulips, pastels; romance: roses, peonies) "
+                "and occasion-specific details (e.g., elegant wrapping).\n"
+                "SUGGEST complementary items: vases, chocolates, cards. "
+                "Also provide care instructions for long-lasting enjoyment.\n"
+                "STRICTLY Confirm all order details before finalizing: flowers, colors, "
+                "delivery address, timing\n"
+                "STRICTLY Collect complete contact information for order updates\n"
+                "STRICTLY Provide ORDER CONFIRMATION with ESTIMATED DELIVERY TIMES\n"
+                "OFFER MULTIPLE PAYMENT OPTIONS (e.g., card, cash, online) and confirm SECURE PROCESSING.\n"
+                "STRICTLY If you are unsure about a request, ask clarifying questions "
+                "to ensure you understand before responding."
+            ),
+        },
+    ]
+
+    context = OpenAILLMContext(messages)
+
+    # Comment out the below line when enabling Speculative Speech Processing
+    context_aggregator = llm.create_context_aggregator(context)
+
+    # Uncomment the below line to enable speculative speech processing
+    # nvidia_context_aggregator = create_nvidia_context_aggregator(context, send_interims=True)
+    # Uncomment the below line to enable speculative speech processing
+    # nvidia_tts_response_cacher = NvidiaTTSResponseCacher()
+
+    pipeline = Pipeline(
+        [
+            transport.input(),  # Websocket input from client
+            stt,  # Speech-To-Text
+            stt_transcript_synchronization,
+            # Comment out the below line when enabling Speculative Speech Processing
+            context_aggregator.user(),
+            # Uncomment the below line to enable speculative speech processing
+            # nvidia_context_aggregator.user(),
+            llm,  # LLM
+            tts,  # Text-To-Speech
+            # Caches TTS responses for coordinated delivery in speculative
+            # speech processing
+            # nvidia_tts_response_cacher,  # Uncomment to enable speculative speech processing
+            tts_transcript_synchronization,
+            transport.output(),  # Websocket output to client
+            context_aggregator.assistant(),
+            # Uncomment the below line to enable speculative speech processing
+            # nvidia_context_aggregator.assistant(),
+        ]
+    )
+
+    task = PipelineTask(
+        pipeline,
+        params=PipelineParams(
+            allow_interruptions=True,
+            enable_metrics=True,
+            enable_usage_metrics=True,
+            send_initial_empty_metrics=True,
+            start_metadata={"stream_id": pipeline_metadata.stream_id},
+        ),
+    )
+
+    @transport.event_handler("on_client_connected")
+    async def on_client_connected(transport, client):
+        # Kick off the conversation.
+        messages.append({"role": "system", "content": "Please introduce yourself to the user."})
+        await task.queue_frames([LLMMessagesFrame(messages)])
+
+    return task
+
+
+app = FastAPI()
+app.include_router(websocket_router)
+runner = ACEPipelineRunner.create_instance(pipeline_callback=create_pipeline_task)
+app.mount("/static", StaticFiles(directory=os.getenv("STATIC_DIR", "../static")), name="static")
+
+if __name__ == "__main__":
+    uvicorn.run("bot:app", host="0.0.0.0", port=8100, workers=4)

examples/speech-to-speech/docker-compose.yml

@@ -0,0 +1,94 @@
+name: voice-agent
+
+services:
+  riva-tts-magpie:
+    image: nvcr.io/nim/nvidia/magpie-tts-multilingual:latest
+    environment:
+      - NGC_API_KEY=${NVIDIA_API_KEY}
+      - NIM_HTTP_API_PORT=9000
+      - NIM_GRPC_API_PORT=50051
+    ports:
+      - "19000:9000"
+      - "50151:50051"
+    volumes:
+      - nim_cache:/opt/nim/.cache
+    shm_size: 8GB
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ['0']
+              capabilities: [gpu]
+
+  riva-asr-parakeet:
+    image: nvcr.io/nim/nvidia/parakeet-1-1b-ctc-en-us:latest
+    environment:
+      - NGC_API_KEY=${NVIDIA_API_KEY}
+      - NIM_HTTP_API_PORT=9001
+      - NIM_GRPC_API_PORT=50052
+      - NIM_TAGS_SELECTOR=mode=str,vad=silero
+    ports:
+      - "19001:9001"
+      - "50152:50052"
+    volumes:
+      - nim_cache:/opt/nim/.cache
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ['1']
+              capabilities: [gpu]
+
+  nvidia-llm:
+    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
+    environment:
+      - NGC_API_KEY=${NVIDIA_API_KEY}
+      - NIM_HTTP_API_PORT=8000
+    ports:
+      - "18000:8000"
+    volumes:
+      - nim_cache:/opt/nim/.cache
+    shm_size: 16GB
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              device_ids: ['2']
+              capabilities: [gpu]
+
+  python-app:
+    build:
+      context: ../../
+      dockerfile: examples/speech-to-speech/Dockerfile
+    
+    ports:
+      - "8100:8100"
+    environment:
+      - NVIDIA_API_KEY=${NVIDIA_API_KEY}
+
+      - RIVA_ASR_URL=riva-asr-parakeet:50052
+      - RIVA_TTS_URL=riva-tts-magpie:50051
+      - NVIDIA_LLM_URL=http://nvidia-llm:8000/v1
+
+      - RIVA_ASR_MODEL=parakeet-1.1b-en-US-asr-streaming-silero-vad-asr-bls-ensemble
+      - RIVA_TTS_MODEL=magpie_tts_ensemble-Magpie-Multilingual
+      - NVIDIA_LLM_MODEL=meta/llama-3.1-8b-instruct
+
+      - RIVA_ASR_LANGUAGE=en-US
+      - RIVA_TTS_LANGUAGE=en-US
+      - RIVA_TTS_VOICE_ID=Magpie-Multilingual.EN-US.Sofia 
+      - ZERO_SHOT_AUDIO_PROMPT= # set this only if using a zero-shot TTS model with a custom audio prompt
+
+      - STATIC_DIR=/app/static
+    volumes:
+      - ../static:/app/static
+    depends_on:
+      - riva-tts-magpie
+      - riva-asr-parakeet
+      - nvidia-llm
+
+volumes:
+  nim_cache: 

examples/speech-to-speech/env.example

@@ -0,0 +1,2 @@
+NVIDIA_API_KEY=
+ZEROSHOT_TTS_NVIDIA_API_KEY=

examples/speech-to-speech/pyproject.toml

@@ -0,0 +1,18 @@
+[project]
+name = "speech-to-speech-example"
+version = "0.1.0"
+description = "NVIDIA ACE Pipecat Speech only Examples"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+ "nvidia-pipecat",
+]
+
+[tool.uv.sources]
+torch = { index = "pytorch" }
+nvidia-pipecat = { path = "../../.", editable = true }
+
+[[tool.uv.index]]
+name = "pytorch"
+url = "https://download.pytorch.org/whl/cpu"
+explicit = true

examples/speech_planner/Dockerfile

@@ -0,0 +1,40 @@
+# Base image
+FROM python:3.12-slim
+
+# Image metadata
+LABEL maintainer="NVIDIA"
+LABEL description="Speech Planner example"
+LABEL version="1.0"
+
+# Environment setup
+ENV PYTHONUNBUFFERED=1
+
+# System dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    ffmpeg \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/* \
+    && pip install --no-cache-dir --upgrade pip uv
+
+# App directory setup
+WORKDIR /app
+
+# App files
+COPY pyproject.toml uv.lock \
+     LICENSE README.md NVIDIA_PIPECAT.md \
+     ./
+COPY src/ ./src/
+COPY examples/static/ ./examples/static/
+COPY examples/speech_planner/ ./examples/speech_planner/
+
+# Example app directory
+WORKDIR /app/examples/speech_planner
+
+# Dependencies
+RUN uv sync --frozen
+
+# Port configuration
+EXPOSE 8100
+
+# Start command
+CMD ["uv", "run", "bot.py"]