Packages

class Pool extends Params with HasLabelCol with HasFeaturesCol with HasWeightCol with Logging

CatBoost's abstraction of a dataset.

Features data can be stored in raw (features column has org.apache.spark.ml.linalg.Vector type) or quantized (float feature values are quantized into integer bin values, features column has Array[Byte] type) form.

Raw Pool can be transformed to quantized form using quantize method. This is useful if this dataset is used for training multiple times and quantization parameters do not change. Pre-quantized Pool allows to cache quantized features data and so do not re-run feature quantization step at the start of an each training.

Linear Supertypes
Logging, HasWeightCol, HasFeaturesCol, HasLabelCol, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Pool
  2. Logging
  3. HasWeightCol
  4. HasFeaturesCol
  5. HasLabelCol
  6. Params
  7. Serializable
  8. Serializable
  9. Identifiable
  10. AnyRef
  11. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Pool(data: DataFrame, pairsData: DataFrame)

    Construct Pool from DataFrame also specifying pairs data in an additional DataFrame

    Construct Pool from DataFrame also specifying pairs data in an additional DataFrame

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolWithPairsTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f, 0),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f, 1),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f, 2),
  Row(Vectors.dense(0.23, 0.01, 0.0), "0.0", 0x1L, 1.2f, 3)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
  StructField("sampleId", LongType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val srcPairsData = Seq(
  Row(0x0L, 0, 1),
  Row(0x1L, 3, 2)
)

val srcPairsDataSchema = Seq(
  StructField("groupId", LongType),
  StructField("winnerId", IntegerType),
  StructField("loserId", IntegerType)
)

val pairsDf = spark.createDataFrame(
  spark.sparkContext.parallelize(srcPairsData),
  StructType(srcPairsDataSchema)
)

val pool = new Pool(df, pairsDf)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")
  .setSampleIdCol("sampleId")

pool.data.show()
pool.pairsData.show()
  2. new Pool(data: DataFrame)

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns.

    Construct Pool from DataFrame Call set*Col methods to specify non-default columns. Only features and label columns with "features" and "label" names are assumed by default.

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[4]")
  .appName("PoolTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12", 0x0L, 0.12f),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22", 0x0L, 0.18f),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34", 0x1L, 1.0f)
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType),
  StructField("groupId", LongType),
  StructField("weight", FloatType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)
  .setGroupIdCol("groupId")
  .setWeightCol("weight")

pool.data.show()
  3. new Pool(uid: String, data: DataFrame = null, featuresLayout: TFeaturesLayoutPtr = null, quantizedFeaturesInfo: QuantizedFeaturesInfoPtr = null, pairsData: DataFrame = null, partitionedByGroups: Boolean = false)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. final val baselineCol: Param[String]
  7. def cache(): Pool

    Persist Datasets of this Pool with the default storage level (MEMORY_AND_DISK).

  8. def calcNanModesAndBorders(nanModeAndBordersBuilder: TNanModeAndBordersBuilder, quantizationParams: QuantizationParamsTrait): Unit
    Attributes
    protected
  9. def checkpoint(): Pool

    Returns Pool with eagerly checkpointed Datasets.

  10. def checkpoint(eager: Boolean): Pool

    Returns Pool with checkpointed Datasets.

    Returns Pool with checkpointed Datasets.

    eager

    Whether to checkpoint Datasets immediately

  11. final def clear(param: Param[_]): Pool.this.type
    Definition Classes
    Params
  12. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  13. def copy(extra: ParamMap): Pool
    Definition Classes
    Pool → Params
  14. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  15. def copyWithModifiedData(modifiedData: DataFrame, partitionedByGroups: Boolean = false): Pool

    used to add additional columns to data (for example estimated features) It is impossible to just write an external function for this because copyValues is protected

  16. def count: Long

    returns

    Number of objects in the dataset, similar to the same method of org.apache.spark.sql.Dataset

  17. def createQuantizationSchema(quantizationParams: QuantizationParamsTrait): QuantizedFeaturesInfoPtr
    Attributes
    protected
  18. def createQuantized(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool
    Attributes
    protected
  19. val data: DataFrame
  20. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  21. def ensurePartitionByGroupsIfPresent(): Pool

    ensure that if groups are present data in partitions contains whole groups in consecutive order

  22. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  24. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  25. def explainParams(): String
    Definition Classes
    Params
  26. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  27. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  28. final val featuresCol: Param[String]
    Definition Classes
    HasFeaturesCol
  29. var featuresLayout: TFeaturesLayoutPtr
    Attributes
    protected
  30. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  31. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  32. final def getBaselineCol: String
  33. def getBaselineCount: Int

    returns

    dimension of formula baseline, 0 if no baseline specified

  34. def getCatFeaturesUniqValueCounts: Array[Int]
  35. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  36. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  37. def getEstimatedFeatureCount: Int
  38. def getFeatureCount: Int
  39. def getFeatureNames: Array[String]
  40. final def getFeaturesCol: String
    Definition Classes
    HasFeaturesCol
  41. def getFeaturesLayout: TFeaturesLayoutPtr
  42. final def getGroupIdCol: String
  43. final def getGroupWeightCol: String
  44. final def getLabelCol: String
    Definition Classes
    HasLabelCol
  45. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  46. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  47. final def getSampleIdCol: String
  48. final def getSubgroupIdCol: String
  49. def getTargetType: ERawTargetType
  50. final def getTimestampCol: String
  51. final def getWeightCol: String
    Definition Classes
    HasWeightCol
  52. final val groupIdCol: Param[String]
  53. final val groupWeightCol: Param[String]
  54. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  55. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  56. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  57. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  58. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  59. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  60. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  61. def isQuantized: Boolean
  62. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  63. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  64. final val labelCol: Param[String]
    Definition Classes
    HasLabelCol
  65. def localCheckpoint(): Pool

    Returns Pool with eagerly locally checkpointed Datasets.

  66. def localCheckpoint(eager: Boolean): Pool

    Returns Pool with locally checkpointed Datasets.

    Returns Pool with locally checkpointed Datasets.

    eager

    Whether to checkpoint Datasets immediately

  67. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  68. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  69. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  70. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  71. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  72. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  73. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  74. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  75. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  76. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  77. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  78. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  79. def mapQuantizedPartitions[R](selectedColumns: Seq[String], includeEstimatedFeatures: Boolean, includePairsIfPresent: Boolean, dstColumnNames: Array[String], dstRowLength: Int, f: (TDataProviderPtr, TDataProviderPtr, ArrayBuffer[Array[Any]], TLocalExecutor) ⇒ Iterator[R])(implicit arg0: Encoder[R], arg1: ClassTag[R]): Dataset[R]

    Map over partitions for quantized Pool

  80. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  81. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  82. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  83. def pairsCount: Long

    returns

    Number of pairs in the dataset

  84. val pairsData: DataFrame
  85. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  86. val partitionedByGroups: Boolean
  87. def persist(): Pool

    Persist Datasets of this Pool with the default storage level (MEMORY_AND_DISK).

  88. def persist(storageLevel: StorageLevel): Pool

    Returns Pool with Datasets persisted with the given storage level.

  89. def quantize(quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Pool

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is useful if QuantizedFeaturesInfo with data for quantization (borders and nan modes) has already been computed. Used, for example, to quantize evaluation datasets after the training dataset has been quantized.

  90. def quantize(quantizationParams: QuantizationParamsTrait = new QuantizationParams()): Pool

    Create Pool with quantized features from Pool with raw features

    Create Pool with quantized features from Pool with raw features

    Example: 

    1. val spark = SparkSession.builder()
  .master("local[*]")
  .appName("QuantizationTest")
  .getOrCreate();

val srcData = Seq(
  Row(Vectors.dense(0.1, 0.2, 0.11), "0.12"),
  Row(Vectors.dense(0.97, 0.82, 0.33), "0.22"),
  Row(Vectors.dense(0.13, 0.22, 0.23), "0.34")
)

val srcDataSchema = Seq(
  StructField("features", SQLDataTypes.VectorType),
  StructField("label", StringType)
)

val df = spark.createDataFrame(spark.sparkContext.parallelize(srcData), StructType(srcDataSchema))

val pool = new Pool(df)

val quantizedPool = pool.quantize(new QuantizationParams)
val quantizedPoolWithTwoBinsPerFeature = pool.quantize(new QuantizationParams().setBorderCount(1))
quantizedPool.data.show()
quantizedPoolWithTwoBinsPerFeature.data.show()
  91. def quantizeForModelApplication[Model <: PredictionModel[Vector, Model]](model: CatBoostModelTrait[Model]): Pool

    Create Pool with quantized features from Pool with raw features.

    Create Pool with quantized features from Pool with raw features. This variant of the method is used when we want to apply CatBoostModel on Pool

  92. val quantizedFeaturesInfo: QuantizedFeaturesInfoPtr
  93. def repartition(partitionCount: Int, byGroupColumnsIfPresent: Boolean = true): Pool

    Repartition data to the specified number of partitions.

    Repartition data to the specified number of partitions. Useful to repartition data to create one partition per executor for training (where each executor gets its' own CatBoost worker with a part of the training data).

  94. def sample(fraction: Double): Pool

    Create subset of this pool with the fraction of the samples (or groups of samples if present)

  95. final val sampleIdCol: Param[String]
  96. final def set(paramPair: ParamPair[_]): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  97. final def set(param: String, value: Any): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  98. final def set[T](param: Param[T], value: T): Pool.this.type
    Definition Classes
    Params
  99. final def setBaselineCol(value: String): Pool.this.type
  100. final def setDefault(paramPairs: ParamPair[_]*): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  101. final def setDefault[T](param: Param[T], value: T): Pool.this.type
    Attributes
    protected
    Definition Classes
    Params
  102. def setFeaturesCol(value: String): Pool

  103. final def setGroupIdCol(value: String): Pool.this.type
  104. final def setGroupWeightCol(value: String): Pool.this.type
  105. def setLabelCol(value: String): Pool

  106. final def setSampleIdCol(value: String): Pool.this.type
  107. final def setSubgroupIdCol(value: String): Pool.this.type
  108. final def setTimestampCol(value: String): Pool.this.type
  109. def setWeightCol(value: String): Pool

  110. final val subgroupIdCol: Param[String]
  111. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  112. final val timestampCol: Param[String]
  113. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  114. val uid: String
    Definition Classes
    Pool → Identifiable
  115. def unpersist(blocking: Boolean): Pool

    Mark Datasets of this Pool as non-persistent, and remove all blocks for them from memory and disk.

    Mark Datasets of this Pool as non-persistent, and remove all blocks for them from memory and disk.

    blocking

    Whether to block until all blocks are deleted.

  116. def unpersist(): Pool

    Mark Datasets of this Pool as non-persistent, and remove all blocks for them from memory and disk.

  117. def updateCatFeaturesInfo(isInitialization: Boolean, quantizedFeaturesInfo: QuantizedFeaturesInfoPtr): Unit
    Attributes
    protected
  118. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  119. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  120. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  121. final val weightCol: Param[String]
    Definition Classes
    HasWeightCol

Inherited from Logging

Inherited from HasWeightCol

Inherited from HasFeaturesCol

Inherited from HasLabelCol

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Caching and Persistence

setParam

Ungrouped