README.md

---
base_model:
- CompVis/stable-diffusion-v1-4
- RiddleHe/SD14_pathology_lora
library_name: diffusers
license: creativeml-openrail-m
inference: true
tags:
- stable-diffusion
- stable-diffusion-diffusers
- text-to-image
- diffusers
- controlnet
- diffusers-training
---

<!-- This model card has been generated automatically according to the information the training script had access to. You
should probably proofread and complete it, then remove this comment. -->


# controlnet-RiddleHe/SD14_pathology_controlnet

These are controlnet weights trained on CompVis/stable-diffusion-v1-4 with new type of conditioning.
You can find some example images below.

prompt: A histopathology image of breast cancer tissue.
![images_0)](./imgs/controlnet_validation_1.png)
prompt: An image of breast cancer histopathology with detailed cellular structures.
![images_1)](./imgs/controlnet_validation_2.png)



## Intended uses & limitations

#### How to use

```python
controlnet = ControlNetModel.from_pretrained("RiddleHe/SD14_pathology_controlnet", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "RiddleHe/SD14_pathology_base", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.to('cuda')

prompt = "A histopathology image of breast cancer tissue."
mask = mask.convert("RGB")  # Provide a mask

generator = torch.Generator(device='cuda').manual_seed(42)

with torch.no_grad():
  out = pipe(prompt, image=mask, num_inference_steps=70, num_images_per_prompt=3, generator=generator).images
```

#### Limitations and bias

[TODO: provide examples of latent issues and potential remediations]

## Training details

The model is trained on 28216 image-mask pairs from the BRCA breast cancer dataset. Input is mask and output is image. 

Mask is a single channel image with integer values from 0 to 21 representing 22 classes, eg. 1 representing tumor, 2 representing stroma.