Update README.md

README.md

@@ -5,9 +5,32 @@ base_model:
 - google/siglip-so400m-patch14-384
 pipeline_tag: image-feature-extraction
 ---
-<div align="center">
 
-# Scaling Vision Pre-Training to 4K Resolution
+## Description: <br>
+
+PS3-1.5K-SigLIP is a vision encoder that extracts visual features from images of up to 1.5K resolution.
+
+This model is for research and development only.
+
+### License/Terms of Use: <br> 
+
+NVIDIA license (see https://huggingface.co/nvidia/PS3-1.5K-SigLIP/blob/main/LICENSE.md)
+
+### Deployment Geography:
+
+Global
+
+### Use Case: <br>
+
+The model is used for extracting visual features from high-resolution images.
+
+### Release Date:  <br>
+
+Huggingface [05/30/2025] via [https://huggingface.co/nvidia/PS3-1.5K-SigLIP] <br> 
+
+## Reference(s):
+
+The model is from the paper [Scaling Vision Pre-Training to 4K Resolution](https://arxiv.org/abs/2503.19903). Useful links:
 
 [![website](https://img.shields.io/badge/website-76b900?style=for-the-badge&logo=safari&labelColor=555555)](https://nvlabs.github.io/PS3/)
 [![Arxiv](https://img.shields.io/badge/Arxiv-b31b1b?style=for-the-badge&logo=arxiv&labelColor=555555)](https://arxiv.org/abs/2503.19903)
@@ -15,31 +38,50 @@ pipeline_tag: image-feature-extraction
 [![PS3 Models](https://img.shields.io/badge/PS3%20Models%20-ffd21e?style=for-the-badge&logo=huggingface&labelColor=555555)](https://huggingface.co/collections/nvidia/ps3-scaling-vision-pre-training-to-4k-resolution-682d0535b61c07afd45242e9)
 [![VILA-HD Models](https://img.shields.io/badge/VILA--HD%20Models%20-ffd21e?style=for-the-badge&logo=huggingface&labelColor=555555)](https://huggingface.co/collections/nvidia/ps3-scaling-vision-pre-training-to-4k-resolution-682d0535b61c07afd45242e9)
 [![PS3 Code](https://img.shields.io/badge/PS3%20Code%20-181717?style=for-the-badge&logo=github&labelColor=555555)](https://github.com/NVlabs/PS3)
-[![VILA-HD Code](https://img.shields.io/badge/VILA--HD%20Code%20-181717?style=for-the-badge&logo=github&labelColor=555555)](https://github.com/NVlabs/VILA/tree/main/vila_hd)
-
-<div style="font-family: charter;">
-  <a href="https://bfshi.github.io" target="_blank" style="color: #6f6f6f; text-decoration: none;">Baifeng Shi</a><sup style="font-size: 0.6em;">1,2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://sites.google.com/site/boyilics/home" target="_blank" style="color: #6f6f6f; text-decoration: none;">Boyi Li</a><sup style="font-size: 0.6em;">1,2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://han-cai.github.io/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Han Cai</a><sup style="font-size: 0.6em;">2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://scholar.google.com/citations?user=OI7zFmwAAAAJ&hl=en/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Yao Lu</a><sup style="font-size: 0.6em;">2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://sifeiliu.net/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Sifei Liu</a><sup style="font-size: 0.6em;">2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://research.nvidia.com/person/marco-pavone" target="blank" style="color: #6f6f6f; text-decoration: none;">Marco Pavone</a><sup style="font-size: 0.6em;">2</sup>
-  <br>
-  <a href="https://jankautz.com/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Jan Kautz</a><sup style="font-size: 0.6em;">2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://hanlab.mit.edu/songhan/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Song Han</a><sup style="font-size: 0.6em;">2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://people.eecs.berkeley.edu/~trevor/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Trevor Darrell</a><sup style="font-size: 0.6em;">1</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://www.pmolchanov.com/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Pavlo Molchanov</a><sup style="font-size: 0.6em;">2</sup>&nbsp;&nbsp;&nbsp;
-  <a href="https://hongxu-yin.github.io/" target="_blank" style="color: #6f6f6f; text-decoration: none;">Hongxu Yin</a><sup style="font-size: 0.6em;">2</sup>
-  <br>
-  </a><sup style="font-size: 0.6em;">1</sup> UC Berkeley&nbsp;&nbsp;&nbsp;
-  </a><sup style="font-size: 0.6em;">2</sup> NVIDIA&nbsp;&nbsp;&nbsp;
-</div>
-
-</div>
-
-<hr style="border: 2px solid gray;"></hr>
 
 
+## Model Architecture:
+**Architecture Type:** Neural Network
+
+**Network Architecture:** Vision Transformer designed for high-resolution images 
+
+This model was developed based on [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384). Please see training designs in the paper.
+ 
+
+## Input: <br>
+**Input Type(s):** Image <br>
+**Input Format:** Red, Green, Blue (RGB) <br>
+**Input Parameters:** 2D <br>
+**Other Properties Related to Input:** Image resolutions up to 1512*1512. <br>
+
+## Output: <br>
+**Output Type(s):** Embeddings <br>
+**Output Format:** Tensor <br>
+**Output Parameters:** 1D <br>
+**Other Properties Related to Output:** Downstream model required to leverage image features <br>
+
+Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA’s hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions. <br> 
+
+## Software Integration:
+**Runtime Engine(s):** 
+N/A <br> 
+
+**Supported Hardware Microarchitecture Compatibility:** <br>
+NVIDIA Ampere <br>
+NVIDIA Blackwell <br>
+NVIDIA Jetson  <br>
+NVIDIA Hopper <br>
+
+**Preferred/Supported Operating System(s):**
+Linux <br>
+Linux 4 Tegra <br>
+QNX  <br>
+Windows <br>
+
+## Model Version(s): 
+
+v1.0 - Initial release
+
 ## Pre-Trained Models
 
 ### PS3 models
@@ -49,7 +91,31 @@ pipeline_tag: image-feature-extraction
 | PS3-1.5K-SigLIP | 1512 * 1512    | [nvidia/PS3-1.5K-SigLIP](https://huggingface.co/nvidia/PS3-1.5K-SigLIP) |
 | PS3-4K-SigLIP   | 3780 * 3780    | [nvidia/PS3-4K-SigLIP](https://huggingface.co/nvidia/PS3-4K-SigLIP)     |
 
-<hr style="border: 2px solid gray;"></hr>
+## Training Datasets: <br>   
+
+75M images <br>
+
+1 dataset that's built based on:
+- SA-1B (https://ai.meta.com/datasets/segment-anything/)
+- IDL (https://huggingface.co/datasets/pixparse/idl-wds)
+   
+Training: 100% <br> 
+
+## Training Dataset:
+
+**Link:**
+We used the following dataset during developing PS3:
+- SA-1B (https://ai.meta.com/datasets/segment-anything/)
+- IDL (https://huggingface.co/datasets/pixparse/idl-wds)
+
+**Data Collection Method by dataset:**  <br>
+Automated
+
+**Labeling Method by dataset:**  <br>
+Automated
+
+**Properties (Quantity, Dataset Descriptions, Sensor(s)):**  <br>
+75M images with resolution up to 4Kx4K.
 
 ## Performance
 
@@ -67,6 +133,10 @@ See Table 1 in the paper for full results.
 | SigLIP + S2         |                                                                         | 3780           | 18225            | OOM     | OOM     | OOM    | OOM     | OOM      | OOM     | OOM         | OOM  |
 | **PS3-4K-SigLIP**   | [nvidia/PS3-4K-SigLIP](https://huggingface.co/nvidia/PS3-4K-SigLIP)     | 3780           | 3840             | 69.8    | 70.9    | 79.1   | 40.5    | 543      | 67.8    | 64.7        | 63.9 |
 
+## Inference:
+**Acceleration Engine:** N/A <br>
+**Test Hardware:** <br>
+The model is tested on NVIDIA A100 GPU.
 
 ## Installation
 
@@ -75,16 +145,13 @@ Install through pip to use PS3 out of the box.
 pip install ps3-torch
 ```
 
-If you would like to make changes to the PS3 code, clone this repo and install in editable mode.
+If you would like to make changes to the PS3 code, go to [PS3 repository](https://github.com/NVlabs/PS3), clone the repo, and install in editable mode.
 ```bash
 cd PS3
 pip install -e .
 ```
 
-<hr style="border: 2px solid gray;"></hr>
-
-
-## Quick Start
+## Inference - Quick Start
 
 Here we show example usage including
 - loading the model
@@ -111,7 +178,7 @@ x = processor(image)["pixel_values"][0].unsqueeze(0).cuda()
 
 ### 2. Encode High-Res Image with Bottom-Up Selection
 
-PS3 can select important high-res patches baed on visual saliency and encode those patches.
+PS3 can select important high-res patches based on visual saliency and encode those patches.
 
 **You can encode the whole high-res image using PS3.**
 ```python
@@ -132,9 +199,9 @@ outs = vision_model(x, num_look_close=2)
 features = outs.last_hidden_state
 print(features.shape)  # (1, 5849, 1152)
 ```
-In this example, it only runs the high-res selection and encoding for twice.
+In this example, it only runs the high-res selection and encoding twice.
 
-Note that PS3 processes at most 2560 high-res patches at a time. Then running high-res selection and encoding for twice gives us 2560 * 2 = 5120 high-res tokens. There is also 729 low-res tokens. That gives us 729 + 5120 = 5849 tokens in total.
+Note that PS3 processes at most 2560 high-res patches at a time. Then running high-res selection and encoding twice gives us 2560 * 2 = 5120 high-res tokens. There is also 729 low-res tokens. That gives us 729 + 5120 = 5849 tokens in total.
 
 **You can also decide how many high-res tokens to process by setting `num_token_look_close`.**
 ```python
@@ -142,7 +209,7 @@ outs = vision_model(x, num_token_look_close=3000)
 features = outs.last_hidden_state
 print(features.shape)  # (1, 3729, 1152)
 ```
-In this example, it only processes 3000 high-res tokens. Note that PS3 only processes 2560 high-res patches at a time. This means it needs to run the high-res selection and encoding for twice, with the first time processing 2560 high-res tokens and the second time processing 440 tokens. In the end it outputs 3729 tokens (3000 high-res + 729 low-res).
+In this example, it only processes 3000 high-res tokens. Note that PS3 only processes 2560 high-res patches at a time. This means it needs to run the high-res selection and encoding twice, with the first time processing 2560 high-res tokens and the second time processing 440 tokens. In the end it outputs 3729 tokens (3000 high-res + 729 low-res).
 
 **Visualize the bottom-up patch selection probabilities.**
 ```python
@@ -255,9 +322,7 @@ This will create a masked feature map `feature_maps` which is a list of feature
 
 
 
-<hr style="border: 2px solid gray;"></hr>
-
-## Inference
+## Inference instructions
 
 [Quick Start](#quick-start) gives some examples of how to use PS3 to encode an image. Below are more detailed explanations of the arguments of model inference.
 
@@ -282,28 +347,29 @@ class PS3VisionModel(PS3PreTrainedModel):
 
 `num_look_close`: how many times to run high-res selection and encoding. PS3 selects and processes 2560 patches each time. If set to `all` then it selects all the high-res patches. If set to `0` then PS3 only returns the low-res features. If set to a larger number than what it needs to encode all the high-res patches, then PS3 will clamp it to the max number needed.
 
-`num_token_look_close`: (optinoal) how many high-res patches to select and process. Similar to `num_look_close` but counts the number of high-res tokens instead of number of running high-res encoding.
+`num_token_look_close`: (optinoal) how many high-res patches to select and process. Similar to `num_look_close` but `num_token_look_close` directly specifies the number of high-res tokens instead of number of running high-res encoding.
 
-`prompt`: (optional) the prompt embedding used to select high-res patches. The prompt embedding can be embedding of some text, or some embedding output by an LLM (see paper). The shape of prompt embedding is (B, C) where B is the batch size (same in `pixel_values`) and C is the embedding dimension (same as PS3 token embedding dimension). If `prompt=None`, then PS3 will select high-res patches based on visual saliency (bottom-up selection).
+`prompt`: (optional) the prompt embedding used to select high-res patches. The prompt embedding can be embedding of some text, or some embedding output by an LLM (see the paper). The shape of prompt embedding is (B, C) where B is the batch size (same in `pixel_values`) and C is the embedding dimension (same as PS3 token embedding dimension). If `prompt=None`, then PS3 will select high-res patches based on visual saliency (bottom-up selection).
 
-`gt_selection_maps`: (optional) the ground truth selection maps for the image. It should be a tensor of 0/1 values with shape (B, h, w). Regions with value 1 means they should be selected. When selectin high-res patches, PS3 will interpolate the `gt_selection_maps` to the same size as the feature map at each scale, prioritize selecting the tokens where the value is 1, and if there's still budget for selecting more tokens, select the rest based on the original selection probability.
+`gt_selection_maps`: (optional) the ground truth selection maps for the image. It should be a tensor of 0/1 values with shape (B, h, w). Regions with value 1 means they should be selected. When selecting high-res patches, PS3 will interpolate the `gt_selection_maps` to the same size as the feature map at each scale, prioritize selecting the tokens where the value is 1, and if there's still budget for selecting more tokens, it will select the rest based on the original selection probability.
 
-`smooth_selection_prob`: (optional) smooth the selectino probability map such that the selected patches won't be distributed too scarcely each time it runs high-res selection. It slightly improves the performance occasinoally when selecting all the patches but usually hurts when selecting parts of the patches.
+`smooth_selection_prob`: (optional) smooth the selection probability map such that the selected patches won't be distributed too scarcely each time it runs high-res selection. It slightly improves the performance occasinoally when selecting all the patches but usually hurts when selecting parts of the patches.
 
-`only_select_first_n_scale`: (optional) only select the first n high-res scales. For example, for PS3-4K model, if `only_select_first_n_scale=2`, then only select and process scales of 756 and 1512, and ignore the scale of 3780.
+`only_select_first_n_scale`: (optional) only select the first n high-res scales. For example, for PS3-4K model, if `only_select_first_n_scale=2`, then it only selects and processes scales of 756 and 1512, and ignores the scale of 3780.
 
 `is_global_text`: (optional) only return the pooled low-res feautres. *It will only be used during pre-training.*
 
 `pool_gt_token_only`: (optional) only pool the tokens inside the gt selection regions. *It will only be used during pre-training.*
 
 
-
-<hr style="border: 2px solid gray;"></hr>
+### Ethical Considerations:
+NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications.  When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse. Please report security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
 
 
 ## More Details
 Please refer to the [PS3 codebase](https://github.com/NVlabs/PS3) for more details.
 
+
 ## Citation
 
 If you find this work useful in your research, please consider citing:
@@ -315,4 +381,6 @@ If you find this work useful in your research, please consider citing:
   journal={arXiv preprint arXiv:2503.19903},
   year={2025}
 }
-```
+```
+
+