nvidia/asset-harvester

Asset Harvester | System Model Card

Paper (coming soon) | Project Page (coming soon) | Code | Model | Data

Description:

Asset Harvester generates 3D assets from a single image or multiple images of vehicles or VRUs extracted from autonomous driving sessions.

It leverages 4 models (see the white paper for architecture) in the process. The AV object Mask2former instance segmentation model is used for image processing when parsing input views from NCore data sessions. The input images are encoded by C-Radio, and the multiview diffusion model, SparseViewDiT, is then used to generate 16 multiview images of the input objects. In cases where camera parameters are not provided, the multiview diffusion model includes a camera pose estimation submodule that predicts camera parameters for the input images. Lastly, an Object TokenGS lifts the images to a 3D asset.

This system is ready for commercial/non-commercial use

🚗 In-the-Wild Examples 🚗

Each row contains the input image, object mask, and a rendering of the harvested 3DGS asset.

1. Vehicles / Trucks / Trailers

2. VRUs

3. Other

License/Terms of Use:

Governing Terms: Use of this model system is governed by the NVIDIA Open Model License Agreement .

Deployment Geography: Global

Release Management:

This system is exposed as a collection of models on HuggingFace and inference scripts on Github.

Automation Level:

Partial Automation

Use Case:

Physical AI developers who are looking to create 3D assets of vehicles or VRUs for either closed-loop simulation or Synthetic Data Generation (SDG).

Known Technical Limitations:

The system is not guaranteed to perform well with occluded objects or objects that are outside of the common distribution. For example, a heavily occluded vehicle can generate a poor or hallucinated 3D asset.

Known Risk(s):

AV and robotics developers should be aware that this model cannot guarantee a 100% success rate. In cases of unsuccessful generation, the output may not possess an accurate real-world representation of the asset and should not be relied upon in safety-critical simulations.

Reference(s): (coming soon)

Asset Harvester: Turning Autonomous Driving Logs into 3D Assets for Simulation

System Architecture

System architecture details described in white paper above.

System Input:

Input Type(s): 1 or more images (up until 4)
Input Format: Red, Green, Blue (RGB)
Input Parameters: Two-Dimensional (2D)
Other Properties Related to Input:

We currently accept up to 4 input images for each object. The resolution of the images are 512x512. The input images are extracted from NVIDIA’s NCore data along w/ other metadata needed for downstream processing:

• Camera orientation of each image
• Camera distance of each image
• Camera field of view of each image
• Bounding box dimensions of each object

System Output:

Output Type(s): Corresponding 3D Gaussian asset to the object in input images
Output Format: Polygon File Format (PLY)
Output Parameters: Three-Dimensional (3D)
Other Properties Related to Output:

A PLY file (3D Gaussian Splatting, 3DGS) contains 3D object data with the following specific components:

• Header: Defines the file structure, including format (ASCII or binary), Gaussian elements, their properties (e.g., position, appearance coefficients, opacity, scale, rotation), and data types (e.g., float, int).
• Gaussian Data: Stores the parameters of each 3D Gaussian as vertex elements: center position (x, y, z), spherical harmonics DC coefficients (f_dc_0, f_dc_1, f_dc_2), opacity, anisotropic scale (scale_0, scale_1, scale_2), and rotation quaternion (rot_0, rot_1, rot_2, rot_3).

Hardware Compatibility:

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Ampere
• NVIDIA Blackwell
• NVIDIA Hopper
• NVIDIA Lovelace

Preferred/Supported Operating Systems: Linux

Hardware Specific Requirements:

The systems can run on a single GPU with an Nvidia GPU with CUDA Compute Capability greater than or equal to 8.0. The following is required:

• GPU performance >= 300 Tflops
• GPU memory size >= 30GB
• GPU memory bandwidth >= 768 GB/s
• System RAM >= 32 GB
• System disk storage >= 100GB
• CPU >= 16 threads x 3GHz

System Version:

Asset_Harvester_GA

Inference:

Engine: Pytorch
Test Hardware: A100, H100

Ethical Considerations:

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Model Card++

Bias

Field	Response
Participation considerations from adversely impacted groups protected classes in model design and testing:	None
Measures taken to mitigate against unwanted bias:	None

Explainability

Field	Response
Intended Domain	Autonomous Driving Simulation
Model Type:	Image-to-3D Asset
Intended Users:	Autonomous Vehicles developers enhancing and improving Neural Reconstruction pipelines.
Output	3D Asset
Describe how the model works	The system takes as an input one or few images, and outputs a corresponding 3D asset
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of	None
Technical Limitations	The system is not guaranteed to perform well with occluded objects or objects that are outside of the common distribution. For example, a heavily occluded vehicle image can generate a poor or hallucinated 3D asset
Verified to have met prescribed NVIDIA quality standards	Yes
Performance Metrics	PSNR (Peak Signal-to-Noise Ratio)
Potential Known Risks	AV and robotics developers should be aware that this model cannot guarantee a 100% success rate. In cases of unsuccessful generation, the output may not possess an accurate real-world representation of the asset and should not be relied upon in safety-critical simulations.
Licensing	Use of this model system is governed by the NVIDIA Open Model License Agreement.

Privacy

Field	Response
Generatable or reverse engineerable personal data?	No
Personal data used to create this model?	Yes
Was consent obtained for any personal data used?	Yes
How often is the dataset reviewed?	Before release
Is a mechanism in place to honor data subject right of access or deletion of personal data?	Yes
If personal data was collected for the development of the model, was it collected directly by NVIDIA?	No
If personal data was collected for the development of the model by NVIDIA, do you maintain or have access to disclosures made to data subjects?	Not Applicable
If personal data was collected for the development of this AI model, was it minimized to only what was required?	Yes
Was data from user interactions with the AI model (e.g. user input and prompts) used to train the model?	No
Is there provenance for all datasets used in training?	Yes
Does data labeling (annotation, metadata) comply with privacy laws?	Yes
Is data compliant with data subject requests for data correction or removal, if such a request was made?	Yes
Applicable Privacy Policy	https://www.nvidia.com/en-us/about-nvidia/privacy-policy/

Safety & Security

Field	Response
Model Application(s):	3D Asset Generation
Describe the life critical impact (if present).	N/A - The system should not be deployed in a vehicle to perform life-critical tasks.
Use Case Restrictions:	Use of this model system is governed by the NVIDIA Open Model License Agreement
Model and dataset restrictions:	The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training