Dataset Format

Each JSON file contains a list of data samples. Every sample uses the global/local token format, where <global> represents the broader context (a scene or an object) and <local> represents a specific part within it (an object or a component).

[
    {
        "conversations": [
            {
                "role": "user",
                "content": "Looking at the scene <global>, explain the appearance of the highlighted object <local> and where it is located."
            },
            {
                "role": "assistant",
                "content": "This object is a tall, narrow cabinet with a light wood-grain finish ..."
            }
        ],
        "global": [
            {
                "id": "036bce3393",
                "feat_path": "data/sonata_feat/036bce3393_down.npz",
                "sample_mask_path": "data/scene_mask/036bce3393_mask_32768.npy"
            }
        ],
        "local": [
            {
                "global_id": "036bce3393",
                "mask_path": "data/train_mask/036bce3393/036bce3393_part_1.npy"
            }
        ],
        "metadata": {
            "tasks": "dense_description",
            "level": "scene-object"
        }
    }
]

Fields

conversations

A list of message objects representing the dialogue. The model is trained to predict all assistant turns.

• role: Either "user" or "assistant".
• content: The message text. User messages contain <global> and <local> placeholders that will be replaced with point cloud embeddings during processing.

global

A list of global point cloud entries. Each <global> placeholder in the conversation maps to an entry here (by order).

• id: A unique identifier for this global point cloud.
• feat_path: Path to the .npz feature file containing feat_down (downsampled point features) and inverse (raw-to-downsampled index mapping).
• sample_mask_path: Path to a .npy mask file used for sampling the global features. May be empty for object-level globals.

local

A list of local (masked) point cloud entries. Each <local> placeholder in the conversation maps to an entry here (by order).

• global_id: References the id of the parent global entry that this local region belongs to.
• mask_path: Path to a .npy mask file that selects the specific points within the global point cloud.

metadata

• tasks: The task type (e.g., "dense_description").
• level: The spatial hierarchy level, which determines the semantics of <global> and <local>:

Data Distribution

Files

• GPT_dataset_qwen25_final_train.json — Training set (Qwen2.5 format)
• GPT_dataset_qwen25_final_test.json — Test set (Qwen2.5 format)
• GPT_dataset_qwen3vl_final_train.json — Training set (Qwen3-VL format)
• GPT_dataset_qwen3vl_final_test.json — Test set (Qwen3-VL format)