All-pairs similarity via DIMSUM

We are often interested in finding users, hashtags and ads that are very similar to one another, so they may be recommended and shown to users and advertisers. To do this, we must consider many pairs of items, and evaluate how “similar” they are to one another. We call this the “all-pairs similarity” problem, sometimes known as a “similarity join.” We have developed a new efficient algorithm to so

[yoshi0309]

行同士の類似度を計算する DIMSUM アルゴリズム。総当りするよりも 40% 効率化されるらしい。/ Spark 1.2 から実装済み。(だたし、Java / Scala のみ)