Add model

README.md

@@ -1,3 +1,100 @@
 ---
-license: cc-by-4.0
+language: ja
+license: cc-by-sa-4.0
+library_name: transformers
+datasets:
+  - cc100
+  - mc4
+  - oscar
+  - wikipedia
+  - izumi-lab/cc100-ja
+  - izumi-lab/mc4-ja-filter-ja-normal
+  - izumi-lab/oscar2301-ja-filter-ja-normal
+  - izumi-lab/wikipedia-ja-20230720
+  - izumi-lab/wikinews-ja-20230728
+
+widget:
+- text: 東京大学で[MASK]の研究をしています。 
+
 ---
+
+# DeBERTa V2 base Japanese
+
+This is a [DeBERTaV2](https://github.com/microsoft/DeBERTa) model pretrained on Japanese texts.
+The codes for the pretraining are available at [retarfi/language-pretraining](https://github.com/retarfi/language-pretraining/releases/tag/v2.2.1).
+
+
+## How to use
+
+You can use this model for masked language modeling as follows:
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+tokenizer = AutoTokenizer.from_pretrained("izumi-lab/deberta-v2-base-japanese")
+model = AutoModelForMaskedLM.from_pretrained("izumi-lab/deberta-v2-base-japanese")
+...
+```
+
+
+## Tokenization
+
+The model uses a sentencepiece-based tokenizer, the vocabulary was trained on the Japanese Wikipedia using [sentencepiece](https://github.com/google/sentencepiece).
+
+
+## Training Data
+
+We used the following corpora for pre-training:
+
+- [Japanese portion of CC-100](https://huggingface.co/datasets/izumi-lab/cc100-ja)
+- [Japanese portion of mC4](https://huggingface.co/datasets/izumi-lab/mc4-ja-filter-ja-normal)
+- [Japanese portion of OSCAR2301](izumi-lab/oscar2301-ja-filter-ja-normal)
+- [Japanese Wikipedia as of July 20, 2023](https://huggingface.co/datasets/izumi-lab/wikipedia-ja-20230720)
+- [Japanese Wikinews as of July 28, 2023](https://huggingface.co/datasets/izumi-lab/wikinews-ja-20230728)
+
+
+We pretrained with the corpora mentioned above for 900k steps, and additionally pretrained with the following financial corpora for 100k steps:
+- Summaries of financial results from October 9, 2012, to December 31, 2022
+- Securities reports from February 8, 2018, to December 31, 2022
+- News articles
+
+
+## Training Parameters
+
+learning_rate in parentheses indicate the learning rate for additional pre-training with the financial corpus.
+- learning_rate: 2.4e-4 (6e-5)
+- total_train_batch_size: 2,016
+- max_seq_length: 512
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06
+- lr_scheduler_type: linear schedule with warmup
+- training_steps: 1,000,000
+- warmup_steps: 100,000
+- precision: FP16
+
+
+## Fine-tuning on General NLU tasks
+
+We evaluate our model with the average of five seeds.  
+Other models are from [JGLUE repository](https://github.com/yahoojapan/JGLUE)
+
+
+| Model                         | JSTS             | JNLI      | JCommonsenseQA |
+|-------------------------------|------------------|-----------|----------------|
+|                               | Pearson/Spearman | acc       | acc            |
+| **DeBERTaV2 base**            | **0.890/0.846**  | **0.xxx** | **0.859**      |
+| Waseda RoBERTa base           | 0.913/0.873      | 0.895     | 0.840          |
+| Tohoku BERT base              | 0.909/0.868      | 0.899     | 0.808          |
+
+
+## Citation
+
+TBA
+
+
+## Licenses
+
+The pretrained models are distributed under the terms of the [Creative Commons Attribution-ShareAlike 4.0](https://creativecommons.org/licenses/by-sa/4.0/).
+
+
+## Acknowledgments
+
+This work was supported in part by JSPS KAKENHI Grant Number JP21K12010, and the JST-Mirai Program Grant Number JPMJMI20B1, Japan.

config.json

@@ -0,0 +1,31 @@
+{
+  "architectures": [
+    "DebertaV2ForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-07,
+  "max_position_embeddings": 512,
+  "max_relative_positions": -1,
+  "model_type": "deberta-v2",
+  "norm_rel_ebd": "layer_norm",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "pooler_dropout": 0,
+  "pooler_hidden_act": "gelu",
+  "pooler_hidden_size": 768,
+  "pos_att_type": "p2c|c2p",
+  "position_biased_input": false,
+  "position_buckets": 256,
+  "relative_attention": true,
+  "share_att_key": true,
+  "torch_dtype": "float16",
+  "transformers_version": "4.31.0",
+  "type_vocab_size": 0,
+  "vocab_size": 32000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:596337893657556b383e9813945f6cb3a990f5e0e90530d64b1d49941cd2ca37
+size 542676485

special_tokens_map.json

@@ -0,0 +1,9 @@
+{
+  "bos_token": "[CLS]",
+  "cls_token": "[CLS]",
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "unk_token": "[UNK]"
+}

spm.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8e9cbe24bc1bb25ef87a4371c222666539011d1a749cd4858a88a64771acc1a
+size 804800

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "[CLS]",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "do_lower_case": false,
+  "eos_token": "[SEP]",
+  "mask_token": "[MASK]",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "sp_model_kwargs": {},
+  "split_by_punct": false,
+  "tokenizer_class": "DebertaV2Tokenizer",
+  "unk_token": "[UNK]"
+}