| --- |
| license: apache-2.0 |
| --- |
| |
| # Anime Classifiers |
|
|
| [Training/inference code](https://github.com/city96/CityClassifiers) | [Live Demo](https://huggingface.co/spaces/city96/AnimeClassifiers-demo) |
|
|
|
|
| These are models that predict whether a concept is present in an image. The performance on high resolution images isn't very good, especially when detecting subtle image effects such as noise. This is due to CLIP using a fairly low resolution (336x336/224x224). |
|
|
| To combat this, tiling is used at inference time. The input image is first downscaled to 1536 (shortest edge - See `TF.functional.resize`), then 5 separate 512x512 areas are selected (4 corners + center - See `TF.functional.five_crop`). This helps as the downscale factor isn't nearly as drastic as passing the entire image to CLIP. As a bonus, it also avoids the issues with odd aspect ratios requiring cropping or letterboxing to work. |
|
|
|  |
|
|
| As for the training, it will be detailed in the sections below for the individual classifiers. At first, specialized models will be trained to a relatively high accuracy, building up a high quality but specific dataset in the process. |
|
|
| Then, these models will be used to split/sort each other's the datasets. The code will need to be updated to support one image being part of more than one class, but the final result should be a clean dataset where each target aspect acts as a "tag" rather than a class. |
|
|
| ## Architecture |
|
|
| The base model itself is fairly simple. It takes embeddings from a CLIP model (in this case, `openai/clip-vit-large-patch14`) and expands them to 1024 dimensions. From there, a single block with residuals is followed by a few linear layers which converge down to the final output. |
|
|
| For the classifier models, the final output goes through `nn.Softmax`. |
|
|
| # Models |
|
|
| ## Chromatic Aberration - Anime |
|
|
| ### Design goals |
|
|
| The goal was to detect [chromatic aberration](https://en.wikipedia.org/wiki/Chromatic_aberration?useskin=vector) in images. |
|
|
| For some odd reason, this effect has become a popular post processing effect to apply to images and drawings. While attempting to train an ESRGAN model, I noticed an odd halo around images and quickly figured out that this effect was the cause. This classifier aims to work as a base filter to remove such images from the dataset. |
|
|
| ### Issues |
|
|
| - Seems to get confused by excessive HSV noise |
| - Triggers even if the effect is only applied to the background |
| - Sometimes triggers on rough linework/sketches (i.e. multiple semi-transparent lines overlapping) |
| - Low accuracy on 3D/2.5D with possible false positives. |
|
|
| ### Training |
|
|
| The training settings can be found in the `config/CCAnime-ChromaticAberration-v1.yaml` file (7e-6 LR, cosine scheduler, 100K steps). |
|
|
|  |
|
|
|  |
|
|
|
|
| Final dataset score distribution for v1.16: |
| ``` |
| 3215 images in dataset. |
| 0_reg - 395 |||| |
| 0_reg_booru - 1805 |||||||||||||||||||||| |
| 1_chroma - 515 |||||| |
| 1_synthetic - 500 |||||| |
| |
| Class ratios: |
| 00 - 2200 ||||||||||||||||||||||||||| |
| 01 - 1015 |||||||||||| |
| ``` |
|
|
| Version history: |
|
|
| - v1.0 - Initial test model, dataset is fully synthetic (500 images). Effect added by shifting red/blue channel by a random amount using chaiNNer. |
| - v1.1 - Added 300 images tagged "chromatic_aberration" from gelbooru. Added first 1000 images from danbooru2021 as reg images |
| - v1.2 - Used the newly trained predictor to filter the existing datasets - found ~70 positives in the reg set and ~30 false positives in the target set. |
| - v1.3-v1.16 - Repeatedly ran predictor against various datasets, adding false positives/negatives back into the dataset, sometimes running against the training set to filter out misclassified images as the predictor got better. Added/removed images were manually checked (My eyes hurt). |
| |
| ## Image Compression - Anime |
| |
| ### Design goals |
| |
| The goal was to detect [compression artifacts](https://en.wikipedia.org/wiki/Compression_artifact?useskin=vector) in images. |
| |
| This seems like the next logical step in dataset filtering. The flagged images can either be cleaned up or tagged correctly so the resulting network won't inherit the image artifacts. |
| |
| ### Issues |
| |
| - Low accuracy on 3D/2.5D with possible false positives. |
| |
| ### Training |
| |
| The training settings can be found in the `config/CCAnime-Compression-v1.yaml` file (2.7e-6 LR, cosine scheduler, 40K steps). |
| |
|  |
| |
| The eval loss only uses a single image for each target class, hence the questionable nature of the graph. |
| |
|  |
| |
| |
| Final dataset score distribution for v1.5: |
| ``` |
| 22736 images in dataset. |
| 0_fpl - 108 |
| 0_reg_aes - 142 |
| 0_reg_gel - 7445 ||||||||||||| |
| 1_aes_jpg - 103 |
| 1_fpl - 8 |
| 1_syn_gel - 7445 ||||||||||||| |
| 1_syn_jpg - 40 |
| 2_syn_gel - 7445 ||||||||||||| |
| 2_syn_webp - 0 |
| |
| Class ratios: |
| 00 - 7695 ||||||||||||| |
| 01 - 7596 ||||||||||||| |
| 02 - 7445 ||||||||||||| |
| ``` |
| |
| Version history: |
| |
| - v1.0 - Initial test model, dataset consists of 40 hand picked images and their jpeg compressed counterpart. Compression is done with ChaiNNer, compression rate is randomized. |
| - v1.1 - Added more images by re-filtering the input dataset using the v1 model, keeping only the top/bottom 10%. |
| - v1.2 - Used the newly trained predictor to filter the existing datasets - found ~70 positives in the reg set and ~30 false positives in the target set. |
| - v1.3 - Scraped ~7500 images from gelbooru, filtering for min. image size of at least 3000 and a file size larger than 8MB. Compressed using ChaiNNer as before. |
| - v1.4 - Added webm compression to the list, decided against adding GIF/dithering since it's rarely used nowadays. |
| - v1.5 - Changed LR/step count to better match larger dataset. Added false positives/negatives from v1.4. |
| |
