SmvDataSet

Instance Constructors

new SmvDataSet()

Abstract Value Members

abstract def description(): String
abstract def dsType(): String

DataSet type: could be 4 values, Input, Link, Module, Output
abstract def isEphemeral: Boolean

flag if this module is ephemeral or short lived so that it will not be persisted when a graph is executed.
flag if this module is ephemeral or short lived so that it will not be persisted when a graph is executed. This is quite handy for "filter" or "map" type modules so that we don't force an extra I/O step when it is not needed. By default all modules are persisted unless the flag is overridden to true. Note: the module will still be persisted if it was specifically selected to run by the user.
abstract def requiresDS(): Seq[SmvDataSet]

modules must override to provide set of datasets they depend on.
modules must override to provide set of datasets they depend on. This is no longer the canonical list of dependencies. Internally we should query resolvedRequiresDS for dependencies.

Concrete Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def allDeps: Seq[SmvDataSet]

All dependencies with the dependency hierarchy flattened
lazy val ancestors: Seq[SmvDataSet]
def app: SmvApp
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
def datasetHash(): Int

Hash computed from the dataset, could be overridden to include things other than CRC
def dqm(): SmvDQM

Define the DQM rules, fixes and policies to be applied to this DataSet.
Define the DQM rules, fixes and policies to be applied to this DataSet. See org.tresamigos.smv.dqm, org.tresamigos.smv.dqm.DQMRule, and org.tresamigos.smv.dqm.DQMFix for details on creating rules and fixes.
Concrete modules and files should override this method to define rules/fixes to apply. The default is to provide an empty set of DQM rules/fixes.
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def exportToHive(collector: SmvRunInfoCollector): Serializable

Exports a dataframe to a hive table.
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def fnpart: String

Names the persisted file for the result of this SmvDataSet
Names the persisted file for the result of this SmvDataSet

Definition Classes
SmvDataSet → FilenamePart
def fqn: String

The FQN of an SmvDataSet is its classname for Scala implementations.
The FQN of an SmvDataSet is its classname for Scala implementations.
Scala proxies for implementations in other languages must override this to name the proxied FQN.
def getAncillary[T <: SmvAncillary](anc: T): T

TODO: remove this method as checkDependency replaced this function
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
def instanceValHash(): Int

Hash computed based on instance values of the dataset, such as the timestamp of an input file *
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
val isObjectInShell: Boolean

Objects defined in Spark Shell has class name start with $ *
def metadata(df: DataFrame): SmvMetadata

Can be overridden to supply custom metadata TODO: make SmvMetadata more user friendly or find alternative format for user metadata
def moduleCsvPath(prefix: String = ""): String

Returns the path for the module's csv output
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def persist(dataframe: DataFrame, prefix: String = ""): Unit
def publishHiveSql: Option[String]

An optional sql query to run to publish the results of this module when the --publish-hive command line is used.
An optional sql query to run to publish the results of this module when the --publish-hive command line is used. The DataFrame result of running this module will be available to the query as the "dftable" table. For example: return "insert overwrite table mytable select * from dftable" If this method is not specified, the default is to just create the table specified by tableName() with the results of the module.
def rdd(forceRun: Boolean = false, genEdd: Boolean = app.genEdd, collector: SmvRunInfoCollector): DataFrame

returns the DataFrame from this dataset (file/module).
returns the DataFrame from this dataset (file/module). The value is cached so this function can be called repeatedly. The cache is external to SmvDataSet so that it we will not recalculate the DF even after dynamically loading the same SmvDataSet. If force argument is true, the we skip the cache. Note: the RDD graph is cached and NOT the data (i.e. rdd.cache is NOT called here)
def readFile(path: String, attr: CsvAttributes = CsvAttributes.defaultCsv): DataFrame

Read a dataframe from a persisted file path, that is usually an input data set or the output of an upstream SmvModule.
Read a dataframe from a persisted file path, that is usually an input data set or the output of an upstream SmvModule.
The default format is headerless CSV with '"' as the quote character
def requiresAnc(): Seq[SmvAncillary]
def resolve(resolver: DataSetResolver): SmvDataSet
var resolvedRequiresDS: Seq[SmvDataSet]

fixed list of SmvDataSet dependencies
def runInfo: SmvRunInfo

Returns the run information from this dataset's last run.
Returns the run information from this dataset's last run.
If the dataset has never been run, returns an empty run info with null for its components.
def setTimestamp(dt: DateTime): Unit
def sourceCodeHash(): Int

Hash computed based on the source code of the dataset's class *
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def tableName: String

full name of hive output table if this module is published to hive.
def toString(): String

Definition Classes
SmvDataSet → AnyRef → Any
def urn: URN
def validateMetadata(metadata: SmvMetadata, history: Seq[SmvMetadata]): Option[String]

Override to validate module results based on current and historic metadata.
Override to validate module results based on current and historic metadata. If Some, DQM will fail. Defaults to None.
def verHex: String
def version(): Int

user tagged code "version".
user tagged code "version". Derived classes should update the value when code or data
def versionedFqn: String
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )

Related Doc: package smv

abstract class SmvDataSet extends FilenamePart

Instance Constructors

new SmvDataSet()

Abstract Value Members

abstract def description(): String

abstract def dsType(): String

abstract def isEphemeral: Boolean

abstract def requiresDS(): Seq[SmvDataSet]

Concrete Value Members

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def allDeps: Seq[SmvDataSet]

lazy val ancestors: Seq[SmvDataSet]

def app: SmvApp

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def datasetHash(): Int

def dqm(): SmvDQM

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def exportToHive(collector: SmvRunInfoCollector): Serializable

def finalize(): Unit

def fnpart: String

def fqn: String

def getAncillary[T <: SmvAncillary](anc: T): T

final def getClass(): Class[_]

def hashCode(): Int

def instanceValHash(): Int

final def isInstanceOf[T0]: Boolean

val isObjectInShell: Boolean

def metadata(df: DataFrame): SmvMetadata

def moduleCsvPath(prefix: String = ""): String

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def persist(dataframe: DataFrame, prefix: String = ""): Unit

def publishHiveSql: Option[String]

def rdd(forceRun: Boolean = false, genEdd: Boolean = app.genEdd, collector: SmvRunInfoCollector): DataFrame

def readFile(path: String, attr: CsvAttributes = CsvAttributes.defaultCsv): DataFrame

def requiresAnc(): Seq[SmvAncillary]

def resolve(resolver: DataSetResolver): SmvDataSet

var resolvedRequiresDS: Seq[SmvDataSet]

def runInfo: SmvRunInfo

def setTimestamp(dt: DateTime): Unit

def sourceCodeHash(): Int

final def synchronized[T0](arg0: ⇒ T0): T0

def tableName: String

def toString(): String

def urn: URN

def validateMetadata(metadata: SmvMetadata, history: Seq[SmvMetadata]): Option[String]

def verHex: String

def version(): Int

def versionedFqn: String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

Inherited from FilenamePart

Inherited from AnyRef

Inherited from Any

Ungrouped