SmvHiveTable

Value Members

1. final def !=(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

2. final def ##(): Int

   

   Definition Classes

   AnyRef → Any

3. final def ==(arg0: Any): Boolean

   

   Definition Classes

   AnyRef → Any

4. def allDeps: Seq[SmvDataSet]

   

   All dependencies with the dependency hierarchy flattened

   All dependencies with the dependency hierarchy flattened

   Definition Classes

   SmvDataSet

5. lazy val ancestors: Seq[SmvDataSet]

   

   Definition Classes

   SmvDataSet

6. def app: SmvApp

   

   Definition Classes

   SmvDataSet

7. final def asInstanceOf[T0]: T0

   

   Definition Classes

   Any

8. def clone(): AnyRef

   

   Attributes

   protected[java.lang]

   Definition Classes

   AnyRef

   Annotations

   @throws( ... )

9. def datasetHash(): Int

   

   Hash computed from the dataset, could be overridden to include things other than CRC

   Hash computed from the dataset, could be overridden to include things other than CRC

   Definition Classes

   SmvDataSet

10. def description(): String

    

    Definition Classes

    SmvHiveTable → SmvDataSet

11. def dqm(): SmvDQM

    

    Define the DQM rules, fixes and policies to be applied to this DataSet.

    Define the DQM rules, fixes and policies to be applied to this DataSet. See org.tresamigos.smv.dqm, org.tresamigos.smv.dqm.DQMRule, and org.tresamigos.smv.dqm.DQMFix for details on creating rules and fixes.

    Concrete modules and files should override this method to define rules/fixes to apply. The default is to provide an empty set of DQM rules/fixes.

    Definition Classes

    SmvDataSet

12. def dsType(): String

    

    DataSet type: could be 4 values, Input, Link, Module, Output

    DataSet type: could be 4 values, Input, Link, Module, Output

    Definition Classes

    SmvInputDataSet → SmvDataSet

13. final def eq(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

14. def equals(arg0: Any): Boolean

    

    Definition Classes

    AnyRef → Any

15. def exportToHive(collector: SmvRunInfoCollector): Serializable

    

    Exports a dataframe to a hive table.

    Exports a dataframe to a hive table.

    Definition Classes

    SmvDataSet

16. def finalize(): Unit

    

    Attributes

    protected[java.lang]

    Definition Classes

    AnyRef

    Annotations

    @throws( classOf[java.lang.Throwable] )

17. def fnpart: String

    

    Names the persisted file for the result of this SmvDataSet

    Names the persisted file for the result of this SmvDataSet

    Definition Classes

    SmvDataSet → FilenamePart

18. def fqn: String

    

    The FQN of an SmvDataSet is its classname for Scala implementations.

    The FQN of an SmvDataSet is its classname for Scala implementations.

    Scala proxies for implementations in other languages must override this to name the proxied FQN.

    Definition Classes

    SmvDataSet

19. def getAncillary[T <: SmvAncillary](anc: T): T

    

    TODO: remove this method as checkDependency replaced this function

    TODO: remove this method as checkDependency replaced this function

    Definition Classes

    SmvDataSet

20. final def getClass(): Class[_]

    

    Definition Classes

    AnyRef → Any

21. def hashCode(): Int

    

    Definition Classes

    AnyRef → Any

22. def instanceValHash(): Int

    

    Hash computed based on instance values of the dataset, such as the timestamp of an input file *

    Hash computed based on instance values of the dataset, such as the timestamp of an input file *

    Definition Classes

    SmvDataSet

23. val isEphemeral: Boolean

    

    flag if this module is ephemeral or short lived so that it will not be persisted when a graph is executed.

    flag if this module is ephemeral or short lived so that it will not be persisted when a graph is executed. This is quite handy for "filter" or "map" type modules so that we don't force an extra I/O step when it is not needed. By default all modules are persisted unless the flag is overridden to true. Note: the module will still be persisted if it was specifically selected to run by the user.

    Definition Classes

    SmvInputDataSet → SmvDataSet

24. final def isInstanceOf[T0]: Boolean

    

    Definition Classes

    Any

25. val isObjectInShell: Boolean

    

    Objects defined in Spark Shell has class name start with $ *

    Objects defined in Spark Shell has class name start with $ *

    Definition Classes

    SmvDataSet

26. def metadata(df: DataFrame): SmvMetadata

    

    Can be overridden to supply custom metadata TODO: make SmvMetadata more user friendly or find alternative format for user metadata

    Can be overridden to supply custom metadata TODO: make SmvMetadata more user friendly or find alternative format for user metadata

    Definition Classes

    SmvDataSet

27. def moduleCsvPath(prefix: String = ""): String

    

    Returns the path for the module's csv output

    Returns the path for the module's csv output

    Definition Classes

    SmvDataSet

28. final def ne(arg0: AnyRef): Boolean

    

    Definition Classes

    AnyRef

29. final def notify(): Unit

    

    Definition Classes

    AnyRef

30. final def notifyAll(): Unit

    

    Definition Classes

    AnyRef

31. def persist(dataframe: DataFrame, prefix: String = ""): Unit

    

    Definition Classes

    SmvDataSet

32. def publishHiveSql: Option[String]

    

    An optional sql query to run to publish the results of this module when the --publish-hive command line is used.

    An optional sql query to run to publish the results of this module when the --publish-hive command line is used. The DataFrame result of running this module will be available to the query as the "dftable" table. For example: return "insert overwrite table mytable select * from dftable" If this method is not specified, the default is to just create the table specified by tableName() with the results of the module.

    Definition Classes

    SmvDataSet

33. val query: String

    

34. def rdd(forceRun: Boolean = false, genEdd: Boolean = app.genEdd, collector: SmvRunInfoCollector): DataFrame

    

    returns the DataFrame from this dataset (file/module).

    returns the DataFrame from this dataset (file/module). The value is cached so this function can be called repeatedly. The cache is external to SmvDataSet so that it we will not recalculate the DF even after dynamically loading the same SmvDataSet. If force argument is true, the we skip the cache. Note: the RDD graph is cached and NOT the data (i.e. rdd.cache is NOT called here)

    Definition Classes

    SmvDataSet

35. def readFile(path: String, attr: CsvAttributes = CsvAttributes.defaultCsv): DataFrame

    

    Read a dataframe from a persisted file path, that is usually an input data set or the output of an upstream SmvModule.

    Read a dataframe from a persisted file path, that is usually an input data set or the output of an upstream SmvModule.

    The default format is headerless CSV with '"' as the quote character

    Definition Classes

    SmvDataSet

36. def requiresAnc(): Seq[SmvAncillary]

    

    Definition Classes

    SmvDataSet

37. def requiresDS(): Seq[Nothing]

    

    modules must override to provide set of datasets they depend on.

    modules must override to provide set of datasets they depend on. This is no longer the canonical list of dependencies. Internally we should query resolvedRequiresDS for dependencies.

    Definition Classes

    SmvInputDataSet → SmvDataSet

38. def resolve(resolver: DataSetResolver): SmvDataSet

    

    Definition Classes

    SmvDataSet

39. var resolvedRequiresDS: Seq[SmvDataSet]

    

    fixed list of SmvDataSet dependencies

    fixed list of SmvDataSet dependencies

    Definition Classes

    SmvDataSet

40. def run(df: DataFrame): DataFrame

    

    Method to run/pre-process the input file.

    Method to run/pre-process the input file. Users can override this method to perform file level ETL operations.

    Definition Classes

    SmvInputDataSet

41. def runInfo: SmvRunInfo

    

    Returns the run information from this dataset's last run.

    Returns the run information from this dataset's last run.

    If the dataset has never been run, returns an empty run info with null for its components.

    Definition Classes

    SmvDataSet

42. def setTimestamp(dt: DateTime): Unit

    

    Definition Classes

    SmvDataSet

43. def sourceCodeHash(): Int

    

    Hash computed based on the source code of the dataset's class *

    Hash computed based on the source code of the dataset's class *

    Definition Classes

    SmvDataSet

44. final def synchronized[T0](arg0: ⇒ T0): T0

    

    Definition Classes

    AnyRef

45. val tableName: String

    

    full name of hive output table if this module is published to hive.

    full name of hive output table if this module is published to hive.

    Definition Classes

    SmvHiveTable → SmvDataSet

46. def toString(): String

    

    Definition Classes

    SmvDataSet → AnyRef → Any

47. def urn: URN

    

    Definition Classes

    SmvDataSet

48. val userQuery: String

    

49. def validateMetadata(metadata: SmvMetadata, history: Seq[SmvMetadata]): Option[String]

    

    Override to validate module results based on current and historic metadata.

    Override to validate module results based on current and historic metadata. If Some, DQM will fail. Defaults to None.

    Definition Classes

    SmvDataSet

50. def verHex: String

    

    Definition Classes

    SmvDataSet

51. def version(): Int

    

    user tagged code "version".

    user tagged code "version". Derived classes should update the value when code or data

    Definition Classes

    SmvDataSet

52. def versionedFqn: String

    

    Definition Classes

    SmvDataSet

53. final def wait(): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

54. final def wait(arg0: Long, arg1: Int): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )

55. final def wait(arg0: Long): Unit

    

    Definition Classes

    AnyRef

    Annotations

    @throws( ... )