Skip to main content
Version: 1.3.1.0

API and Operations

HWC exposes HiveWarehouseSession as the primary API for executing Hive operations from Spark.

Create a session​

import com.hortonworks.hwc.HiveWarehouseSession

val hwc = HiveWarehouseSession.session(spark).build()

Catalog operations​

hwc.showDatabases().show(false)
hwc.showTables().show(false)
hwc.describeTable("db.table").show(false)

hwc.createDatabase("db", true)

hwc.createTable("db.table")
  .ifNotExists()
  .column("id", "bigint")
  .column("v", "string")
  .create()

hwc.dropTable("db.table", true, true)
hwc.dropDatabase("db", true, true)

Read and write operations​

// Read
val df = hwc.sql("select * from db.table")

// Write
spark.range(0, 100)
  .selectExpr("id", "concat('v', id) as v")
  .write
  .format("com.hortonworks.spark.sql.hive.llap.HiveWarehouseConnector")
  .option("database", "db")
  .option("table", "table")
  .mode("overwrite")
  .save()

ACID transactions​

Use commitTxn() to finalize Hive ACID transactions managed by the connector:

hwc.commitTxn()

Close the session​

hwc.close()