Finally, a Replacement for BERT: Introducing ModernBERT
โข
674
None defined yet.
backend="onnx"
or backend="openvino"
when initializing a SparseEncoder to get started, but I also included utility functions for optimization, dynamic quantization, and static quantization, plus benchmarks.n-tuple-scores
output format from mine_hard_negatives
gather_across_devices=True
to load in-batch negatives from the other devices too! Essentially a free lunch, pretty big impact potential in my evals.transformers
, and you install trackio
with pip install trackio
, then your experiments will also automatically be tracked locally with trackio. Just open up localhost and have a look at your losses/evals, no logins, no metric uploading.CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2", backend="onnx")
export_optimized_onnx_model
, export_dynamic_quantized_onnx_model
, and export_static_quantized_openvino_model
functions now work with CrossEncoder rerankers, allowing you to optimize (e.g. fusions, gelu approximations, etc.) or quantize (int8 weights) rerankers.mine_hard_negatives
. absolute_margin
ensures that sim(query, negative) < sim(query, positive) - absolute_margin
, i.e. an absolute margin between the negative & positive similarities.relative_margin
ensures that sim(query, negative) < sim(query, positive) * (1 - relative_margin)
, i.e. a relative margin between the negative & positive similarities.