#Oracle scratchpad update#
#Oracle scratchpad full#

If your Hive table uses a SerDe other than the ones that ship with Base CDH5, you’ll need to upload the SerDe into BDD’s EDP JAR file area in HDFS and update some JAR reference files before the import will work, as detailed in Chapter 3 of the Big Data Discovery Data Processing Guide doc.

#Oracle scratchpad full#

You can only load Hive tables, not Hive views, as the Spark loading process only works with full table definitions in the Hive metastore.

There’s a couple of things to bear in-mind when you’re loading data into BDD in this way: The diagram below shows the basic load, profile and enrich ingestion process. If the Hive table has fewer than 1m rows the whole dataset gets loaded in, or the dataset is sampled if the number of Hive rows is greater than 1m. data_processing_CLI -t bdd_test_tweetsīig Data Discovery would then read the Hive table metastore to get the table and column names, datatypes and file location, then spin-up a Spark job to sample, enrich and then load the data into the DGraph engine. For example, to load a Hive table called “bdd_test_tweets” into the DGraph engine using the command-line, you’d enter the commands: Middleware]$ cd BDD1.0/dataprocessing/edp_cli There’s two basic ways to get data into BDD’s DGraph engine you can either use the Data Processing CLI command-line utility to sample, ingest and enrich Hive table data into the DGraph engine, or you can use the web-based data uploader to ingest data from a spreadsheet, text file or similar. I’ll start first with the data ingestion and transformation part of BDD. Tim is going to concentrate on a data analysis and visualization example using a financial/trading dataset, and I’m going to look at at some of the trickier, or less obvious aspects to the BDD development process that we’ve come across putting together PoCs and demos for customers. I’ve been using Oracle Big Data Discovery for a couple of months now, and one of the sessions I’m delivering at this week’s Rittman Mead BI Forum 2015 in Atlanta is on Big Data Discovery Examples from the Field, co-presented with Tim Vlamis from Vlamis Software.