sampleclean.clean.featurize

SimilarityFeaturizer

class SimilarityFeaturizer extends Featurizer

One common type of featurizers are Similarity Featurizers; these are widely used in deduplication and entity resolution workflows. In this case, a Similarity Featurizer will take two rows and calculate a series of values that represent different aspects of "similiarity" between them.

Each aspect will be one of the following similarity metrics:

"BlockDistance", "ChapmanLengthDeviation", "ChapmanMatchingSoundex" "ChapmanMeanLength", "ChapmanOrderedNameCompoundSimilarity", "CosineSimilarity" "DiceSimilarity", "EuclideanDistance", "JaccardSimilarity", "Jaro" "JaroWinkler" ,"Levenshtein", "MatchingCoefficient","MongeElkan" "NeedlemanWunch", "OverlapCoefficient", "QGramsDistance", "SmithWaterman" "SmithWatermanGotoh", "SmithWatermanGotohWindowedAffine", "Soundex" "TagLinkToken"

Annotations
@serializable()
Linear Supertypes
Featurizer, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SimilarityFeaturizer
  2. Featurizer
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SimilarityFeaturizer(colNames: List[String], context: List[String], metrics: List[String])

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. var cols: List[Int]

    Definition Classes
    Featurizer
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def featurize[K, V](rows: Set[Row], params: Map[K, V] = null): (Set[Row], Array[Double])

    This function takes a set of rows and returns a tuple.

    This function takes a set of rows and returns a tuple.

    The first element of the tuple is a set of primary keys and the second element is an array of doubles which are the features (known as a feature vector).

    rows

    A set of Rows. If more than 2, only the first and last will be compared.

    params

    not needed for this implementation.

    Definition Classes
    SimilarityFeaturizerFeaturizer
  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. def getSimilarities(s1: String, s2: String, simMeasures: List[String] = metrics): List[Double]

    This function calculates a list of similarities between two strings.

    This function calculates a list of similarities between two strings. All use white space tokenizer by default and assume equal token weights

    s1

    First string

    s2

    Second string

    simMeasures

    list of allowed similarity measures (see SimilarityFeaturizer )

  15. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  16. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  21. def toString(): String

    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Featurizer

Inherited from AnyRef

Inherited from Any

Ungrouped