Upload folder using huggingface_hub

README.md

@@ -6,206 +6,87 @@ tags:
 - financial-text
 - boilerplate-detection
 - analyst-reports
-- transformers
 pipeline_tag: text-classification
 widget:
-- text: "EEA - The securities and related financial instruments described herein may not be eligible for sale in all jurisdictions or to certain categories of investors."
-  example_title: "Legal Disclaimer"
-- text: "This report contains forward-looking statements that involve risks and uncertainties regarding future events."
-  example_title: "Forward-Looking Statement"
+- text: "The securities and related financial instruments described herein may not be eligible for sale in all jurisdictions or to certain categories of investors."
 - text: "Our revenue increased by 15% compared to last quarter due to strong demand in emerging markets."
-  example_title: "Business Performance"
-- text: "The information contained herein is confidential and proprietary and may not be disclosed without written permission."
-  example_title: "Confidentiality Notice"
-- text: "We launched three innovative products this quarter that exceeded our initial sales projections by 40%."
-  example_title: "Product Update"
+- text: "This report contains forward-looking statements that involve risks and uncertainties."
+- text: "We launched three innovative products this quarter that exceeded our sales projections by 40%."
 ---
 
-# Boilerplate Detection Model for Financial Documents
+# Boilerplate Detection for Financial Text
 
-This model detects boilerplate (formulaic/repetitive) text in financial analyst reports, distinguishing it from substantive business content.
+This model identifies boilerplate (formulaic, repetitive) language in financial documents, distinguishing it from substantive business content. It was developed to preprocess analyst reports for research on corporate culture analysis.
 
 ## Model Description
 
-Developed for analyzing corporate culture discussions in analyst reports by filtering out standardized boilerplate content including legal disclaimers, forward-looking statements, and other formulaic language.
+The model uses a frozen sentence transformer (all-mpnet-base-v2) combined with a lightweight classification head to identify boilerplate text segments. Training data consisted of analyst reports from 2000-2020, where boilerplate examples were identified as frequently repeated segments across reports from the same brokerage house.
 
-### Research Context
+The architecture combines mean-pooled embeddings from the sentence transformer with a simple 3-layer neural network (768 → 16 → 8 → 2) for classification. This approach preserves semantic understanding while learning patterns specific to financial boilerplate language.
 
-This model was developed as part of the research paper "Dissecting Corporate Culture Using Generative AI" to preprocess analyst reports for culture analysis. The model identifies and removes boilerplate segments that would otherwise introduce noise in substantive content analysis.
+## Usage
 
-### Training Methodology
+Since this model uses a custom architecture, you need to use the direct loading approach rather than the pipeline interface:
 
-1. **Data Collection**: 
-   - 2.4 million analyst reports from Thomson One's Investext (2000-2020)
-   - Reports from top 20 brokers by volume analyzed systematically
-
-2. **Training Data**:
-   - **Positive examples (boilerplate)**: Top 10% most frequently repeated segments per broker-year, appearing ≥5 times
-   - **Negative examples**: Randomly selected non-repeated segments
-   - **Dataset**: 547,790 examples (54,779 boilerplate, 493,011 non-boilerplate)
-   - **Split**: 80/10/10 for train/validation/test
-
-3. **Architecture Design**:
-   - **Embedding Layer**: Frozen sentence-transformers/all-mpnet-base-v2
-   - **Pooling**: Mean pooling over token embeddings
-   - **Classification Head**: Lightweight 3-layer MLP (768 → 16 → 8 → 2)
-   - **Strategy**: Frozen embeddings preserve semantic understanding while classification head learns boilerplate patterns
-
-4. **Performance Metrics**:
-   - **Test AUC**: 0.966
-   - **False Positive Rate**: 0.093
-   - **False Negative Rate**: 0.073
-   - **Decision threshold**: 0.22 (median probability)
-
-## Intended Uses
+```python
+import sys
+import huggingface_hub
+from transformers import AutoTokenizer
+import torch
 
-### Primary Use Cases
-- Preprocessing financial analyst reports for content analysis
-- Filtering boilerplate from earnings call transcripts
-- Cleaning regulatory filings for substantive information extraction
-- Preparing financial text for sentiment analysis or topic modeling
+# Load model components
+model_path = huggingface_hub.snapshot_download('maifeng/boilerplate_detection')
+sys.path.insert(0, model_path)
 
-### Out-of-Scope Uses
-- General web content filtering (trained on financial documents)
-- Non-English text classification
-- Real-time streaming applications (optimized for batch processing)
+from modeling_boilerplate import BoilerplateDetector, BoilerplateConfig
 
-## Usage Examples
+# Initialize model
+config = BoilerplateConfig.from_pretrained('maifeng/boilerplate_detection')
+model = BoilerplateDetector.from_pretrained('maifeng/boilerplate_detection')
+tokenizer = AutoTokenizer.from_pretrained('maifeng/boilerplate_detection')
 
-### Using the Transformers Pipeline (Recommended)
+model.eval()
 
-```python
-from transformers import pipeline
-
-# Load the model (requires trust_remote_code=True for custom architecture)
-classifier = pipeline(
-    "text-classification",
-    model="maifeng/boilerplate_detection",
-    trust_remote_code=True,
-    device=0 if torch.cuda.is_available() else -1
-)
-
-# Single text classification
-text = "This report contains forward-looking statements that involve risks and uncertainties."
-result = classifier(text)
-print(result)
-# Output: [{'label': 'BOILERPLATE', 'score': 0.9987}]
-
-# Batch classification for efficiency
+# Classify texts
 texts = [
-    "Revenue increased by 15% this quarter driven by strong product demand.",
     "The securities described herein may not be eligible for sale in all jurisdictions.",
-    "Our new AI initiative has reduced operational costs by 30%.",
-    "Past performance is not indicative of future results.",
+    "Revenue increased by 15% this quarter due to strong market demand.",
+    "This report contains forward-looking statements involving risks.",
+    "Our new product line exceeded initial sales expectations significantly."
 ]
 
-results = classifier(texts, batch_size=32)
-for text, result in zip(texts, results):
-    label = result['label']
-    score = result['score']
-    print(f"{'[BOILERPLATE]' if label == 'BOILERPLATE' else '[CONTENT]    '} "
-          f"(confidence: {score:.1%}) {text[:60]}...")
-```
-
-### Direct Model Usage
-
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
-
-# Load model and tokenizer with trust_remote_code
-model = AutoModel.from_pretrained(
-    "maifeng/boilerplate_detection",
-    trust_remote_code=True
-)
-tokenizer = AutoTokenizer.from_pretrained("maifeng/boilerplate_detection")
-
-# Prepare input
-texts = ["Your text here", "Another example"]
-inputs = tokenizer(
-    texts,
-    padding=True,
-    truncation=True,
-    max_length=512,
-    return_tensors="pt"
-)
-
-# Get predictions
-model.eval()
-with torch.no_grad():
-    outputs = model(**inputs)
-    probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)
-    
-# Process results
-for i, text in enumerate(texts):
-    probs = probabilities[i].numpy()
-    label = "BOILERPLATE" if probs[1] > 0.5 else "NOT_BOILERPLATE"
-    confidence = probs[1] if label == "BOILERPLATE" else probs[0]
-    print(f"{label}: {confidence:.2%} - {text[:50]}...")
-```
-
-### Integration in Document Processing Pipeline
-
-```python
-def filter_boilerplate(documents, threshold=0.5):
-    """Filter out boilerplate segments from documents"""
-    classifier = pipeline(
-        "text-classification",
-        model="maifeng/boilerplate_detection",
-        trust_remote_code=True
-    )
+results = []
+for text in texts:
+    inputs = tokenizer(text, return_tensors='pt', truncation=True, max_length=512)
     
-    results = classifier(documents, batch_size=32)
+    with torch.no_grad():
+        outputs = model(**inputs)
+        probs = torch.nn.functional.softmax(outputs.logits, dim=-1)[0]
     
-    filtered_docs = []
-    for doc, result in zip(documents, results):
-        if result['label'] == 'NOT_BOILERPLATE' or result['score'] < threshold:
-            filtered_docs.append(doc)
+    label = 'BOILERPLATE' if probs[1] > 0.5 else 'NOT_BOILERPLATE'
+    confidence = probs[1].item() if label == 'BOILERPLATE' else probs[0].item()
     
-    return filtered_docs
+    results.append({'text': text, 'label': label, 'confidence': confidence})
 
-# Example usage
-analyst_reports = [...]  # Your document segments
-substantive_content = filter_boilerplate(analyst_reports)
-print(f"Retained {len(substantive_content)}/{len(analyst_reports)} segments")
+for result in results:
+    print(f"{result['label']:>15}: {result['confidence']:.1%} - {result['text'][:60]}...")
 ```
 
 ## Model Limitations
 
-1. **Domain Specificity**: Optimized for financial analyst reports; performance may degrade on other document types
-2. **Temporal Bias**: Trained on 2000-2020 data; newer boilerplate patterns may not be recognized
-3. **Language**: English-only model
-4. **Context Window**: Maximum 512 tokens per segment
-5. **Binary Classification**: Does not distinguish between types of boilerplate
-
-## Ethical Considerations
-
-- **Transparency**: Users should understand that substantive content may occasionally be misclassified as boilerplate
-- **Bias**: Training data from top brokers may not represent all financial communication styles
-- **Use Case**: Should not be used as sole method for regulatory compliance or legal document analysis
+This model is specifically trained on financial analyst reports from 2000-2020 and performs best on similar English-language financial documents. It may not generalize well to other domains or document types. The model processes text segments up to 512 tokens and provides binary classification only.
 
 ## Citation
 
 ```bibtex
-@article{mai2024dissecting,
+@article{li2025dissecting,
   title={Dissecting Corporate Culture Using Generative AI},
-  author={Mai, Feng and others},
-  journal={Working Paper},
-  year={2024}
+  author={Li, Kai and Mai, Feng and Shen, Rui and Yang, Chelsea and Zhang, Tengfei},
+  journal={Review of Financial Studies},
+  year={2025}
 }
 ```
 
-## Technical Requirements
-
-- Python 3.7+
-- PyTorch 1.9+
-- Transformers 4.20+
-- CUDA (optional, for GPU acceleration)
-
 ## License
 
-Apache 2.0 - See LICENSE file for details
-
-## Contact
-
-For questions or issues, please open an issue on the [model repository](https://huggingface.co/maifeng/boilerplate_detection).
+Apache 2.0

config.json

@@ -8,6 +8,7 @@
     8
   ],
   "dropout": 0.05,
+  "dtype": "float32",
   "hidden_size": 768,
   "id2label": {
     "0": "NOT_BOILERPLATE",
@@ -18,6 +19,5 @@
     "NOT_BOILERPLATE": 0
   },
   "model_type": "boilerplate",
-  "torch_dtype": "float32",
-  "transformers_version": "4.53.3"
+  "transformers_version": "4.56.1"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d30e88acc6da21ba6c12a67e26c2fdd11e87976c0c3f1ae06c773ee5f19bbfe2
+oid sha256:ae32f779630e735df1f705d9b7d4743541c6d7f604d00b61e59e44cad7c25dca
 size 438020320