shell

Value Members

1. object ShellCmd

   

   Provide functions for the interactive shell

   Provide functions for the interactive shell

   In SMV's tools/conf/smv_shell_init.scala or project's conf/shell_init.scala add

   import org.tresamigos.smv.shell._

2. def ancestors(dsName: String): Unit

   

3. def ancestors(ds: SmvDataSet): Unit

   

   list all ancestors of a dataset ancestors are datasets current dataset depends on, directly or in-directly, even include datasets from other stages

4. def ddf(fqn: String): Unit

   

5. def ddf(ds: SmvDataSet): Unit

   

6. def descendants(dsName: String): Unit

   

7. def descendants(ds: SmvDataSet): Unit

   

8. def df(ds: SmvDataSet): DataFrame

   

   Resolve SmvDataSet

   Resolve SmvDataSet

   ds

   an SmvDataSet

   returns

   result DataFrame

9. def exportToHive(dsName: String): Serializable

   

   Export dataset's running result to a Hive table

10. def graph(ds: SmvDataSet): Unit

    

    take a DS, print in-stage dependency of that DS

11. def graph(): Unit

    

    take no parameter, print stages and inter-stage links

12. def graph(stageName: String): Unit

    

    take a stage name and print all DS in this stage, without unused input DS

13. def graphStage: Unit

    

14. def help: Unit

    

    list all the smv-shell commands

15. def ls: Unit

    

    list all the datasets in the entire project

16. def ls(stageName: String): Unit

    

    list all datasets in a stage

    list all datasets in a stage

    stageName

    could be the FQN or just the basename

17. def lsDead: Unit

    

    list dead datasets in the entire project

18. def lsDead(stageName: String): Unit

    

    list dead datasets in a stage dead dataset is defined as "no contribution to the Output modules of the stage"

    list dead datasets in a stage dead dataset is defined as "no contribution to the Output modules of the stage"

    stageName

    could be the FQN or the basename

19. def lsDeadLeaf: Unit

    

    list deadLeaf datasets in the entire project

20. def lsDeadLeaf(stageName: String): Unit

    

    list deadLeaf datasets in a stage deadLeaf dataset is defined as "no modules in the stage depend on it, excluding Output modules" Note: a deadLeaf dataset must be dead, but some dead datasets are Not deadLeafs

    list deadLeaf datasets in a stage deadLeaf dataset is defined as "no modules in the stage depend on it, excluding Output modules" Note: a deadLeaf dataset must be dead, but some dead datasets are Not deadLeafs

    stageName

    could be the FQN or the basename

21. def lsStage: Unit

    

    list all the stages

22. def now(): Unit

    

    Print current time

23. def openCsv(path: String, ca: CsvAttributes = null, parserCheck: Boolean = false): DataFrame

    

    Read in a Csv file as DF

24. def openHive(tableName: String): DataFrame

    

    Read in a Hive table as DF

25. def smvDiscoverSchemaToFile(path: String, n: Int = 100000, ca: CsvAttributes = CsvAttributes.defaultCsvWithHeader): Unit

    

    Try best to discover Schema from raw Csv file

    Try best to discover Schema from raw Csv file

    path

    Csv file path and name

    n

    number of records to check for schema discovery, default 100k

    ca

    CsvAttributes, default CsvWithHeader Will save a schema file with postfix ".toBeReviewed" in local directory.