sentence similarity vs text (multi-sentence) similarity

1. sentence similarity

1.1 方法列举

BERT
Universal Sentence Encoder
ELECTRA embedding

1.2 介绍

1.2.1 BERT
With the advancement in language models, representation of sentences into vectors has been getting better lately. That might give some good result in your case. For example, BERT can be used to get the sentence embedding.

Supervised：BERT for sentence similarity if you have labelled set of data

You can use the pre-trained BERT model and you can pass two sentences and you can let the vector obtained at [CLS] pass through a feed forward neural network to decide whether the sentences are similar. This approach can work if you have labelled set of data. If you don’t have, consider the following :

Unsupervised：BERT for single sentence

You pass the variable length sentences to the BERT network and the vector obtained at the token [CLS] becomes the vector for the sentence. You can then use cosine similarity the way you have been using.

1.2.2 Universal Sentence Encoder

https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46808.pdf

1.2.3 ELECTRA: PRE-TRAINING TEXT ENCODERS AS DISCRIMINATORS RATHER THAN GENERATORS

https://arxiv.org/pdf/2003.10555.pdf

a more sample-efficient pre-training task called replaced token detection.

Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not.

• 1.3 实践

Easy sentence similarity with BERT Sentence Embeddings using John Snow Labs NLU：
https://medium.com/spark-nlp/easy-sentence-similarity-with-bert-sentence-embeddings-using-john-snow-labs-nlu-ea078deb6ebf

利用Bert构建句向量并计算相似度：
https://netycc.com/2018/12/05/%E5%88%A9%E7%94%A8bert%E6%9E%84%E5%BB%BA%E5%8F%A5%E5%90%91%E9%87%8F%E5%B9%B6%E8%AE%A1%E7%AE%97%E7%9B%B8%E4%BC%BC%E5%BA%A6/

bert-as-service框架：require only two lines of code to get sentence/token-level encodes.
Finally, bert-as-service uses BERT as a sentence encoder and hosts it as a service via ZeroMQ, allowing you to map sentences into fixed-length representations in just two lines of code.

2. text similarity

2.1 方法：
WDM (for word-level, WDM, for sentence-level, SDM)
Sentence Mover’s Similarity is a variation of Word Mover’s Similarity.

2.2 介绍：

2.2.1 WDM

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to “travel” to reach the embedded words of another document.

For example:

Source: “From Word Embeddings To Document Distances” Paper

WMD can be modified to Sentence Mover’s Distance, comparing how far apart different sentence embeddings are to each other.

2.2.2 SDM

Sentence Mover’s Similarity:
https://homes.cs.washington.edu/~nasmith/papers/clark+celikyilmaz+smith.acl19.pdf