SmvDSWithParser

abstract def description(): String

abstract def dsType(): String

DataSet type: could be 4 values, Input, Link, Module, Output

abstract def isEphemeral: Boolean

flag if this module is ephemeral or short lived so that it will not be persisted when a graph is executed.

abstract def requiresDS(): Seq[SmvDataSet]

modules must override to provide set of datasets they depend on.

final def !=(arg0: Any): Boolean

final def ##(): Int

final def ==(arg0: Any): Boolean

def allDeps: Seq[SmvDataSet]

All dependencies with the dependency hierarchy flattened

lazy val ancestors: Seq[SmvDataSet]

def app: SmvApp

final def asInstanceOf[T0]: T0

def clone(): AnyRef

def datasetHash(): Int

Hash computed from the dataset, could be overridden to include things other than CRC

def dqm(): SmvDQM

Define the DQM rules, fixes and policies to be applied to this DataSet.

def dqmWithTypeSpecificPolicy(userDQM: SmvDQM): SmvDQM

Add parser failure policy to any DataSets that use a parser (e.g.

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def exportToHive(collector: SmvRunInfoCollector): Serializable

Exports a dataframe to a hive table.

val failAtParsingError: Boolean

def finalize(): Unit

def fnpart: String

Names the persisted file for the result of this SmvDataSet

val forceParserCheck: Boolean

def fqn: String

The FQN of an SmvDataSet is its classname for Scala implementations.

def getAncillary[T <: SmvAncillary](anc: T): T

TODO: remove this method as checkDependency replaced this function

final def getClass(): Class[_]

def hashCode(): Int

def instanceValHash(): Int

Hash computed based on instance values of the dataset, such as the timestamp of an input file *

final def isInstanceOf[T0]: Boolean

val isObjectInShell: Boolean

Objects defined in Spark Shell has class name start with $ *

def metadata(df: DataFrame): SmvMetadata

Can be overridden to supply custom metadata TODO: make SmvMetadata more user friendly or find alternative format for user metadata

def moduleCsvPath(prefix: String = ""): String

Returns the path for the module's csv output

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def persist(dataframe: DataFrame, prefix: String = ""): Unit

def publishHiveSql: Option[String]

An optional sql query to run to publish the results of this module when the --publish-hive command line is used.

def rdd(forceRun: Boolean = false, genEdd: Boolean = app.genEdd, collector: SmvRunInfoCollector): DataFrame

returns the DataFrame from this dataset (file/module).

def readFile(path: String, attr: CsvAttributes = CsvAttributes.defaultCsv): DataFrame

Read a dataframe from a persisted file path, that is usually an input data set or the output of an upstream SmvModule.

def requiresAnc(): Seq[SmvAncillary]

def resolve(resolver: DataSetResolver): SmvDataSet

var resolvedRequiresDS: Seq[SmvDataSet]

fixed list of SmvDataSet dependencies

def runInfo: SmvRunInfo

Returns the run information from this dataset's last run.

def setTimestamp(dt: DateTime): Unit

def sourceCodeHash(): Int

Hash computed based on the source code of the dataset's class *

final def synchronized[T0](arg0: ⇒ T0): T0

def tableName: String

full name of hive output table if this module is published to hive.

def toString(): String

def urn: URN

def validateMetadata(metadata: SmvMetadata, history: Seq[SmvMetadata]): Option[String]

Override to validate module results based on current and historic metadata.

def verHex: String

def version(): Int

user tagged code "version".

def versionedFqn: String

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit