스파크 SQL 테이블 생성 방법

스파크 SQL 테이블 생성 방법

1. 스파크 쉘 실행 (cmd창)

   c:\spark-shell

2. hive SQL을 스칼라에서 사용하도록 선언

   val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

3. emp 테이블을 생성하는 명령어

   sqlContext.sql("create table  IF NOT EXISTS emp (empno int, ename string, job string, mgr int, hiredate string, sal int, comm int, deptno int) ROW FORMAT DELIMITED  FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'")

4. emp 테이블에 emp2.txt를 로드(입력)

   sqlContext.sql("LOAD DATA LOCAL INPATH '/home/scott/emp2.txt' INTO TABLE emp")

5. 입력된 데이터 조회SQL

   sql("select * from emp").show()

• 출력된 결과

scala> sql("select * from emp").show()
+-----+------+-------+----+--------+------+----+------+
|empno| ename|    job| mgr|hiredate|   sal|comm|deptno|
+-----+------+-------+----+--------+------+----+------+
|  111|scoot1|manager| 222|20220101| 10000|null|   123|
|  112|scoot2|       |null|20220101| 20000|null|   123|
|  113|scoot3|       |null|20220101| 30000|null|   123|
|  114|scoot4|       |null|20220101| 40000|null|   123|
|  115|scoot5|       |null|20220101| 50000|null|   123|
|  116|scoot6|       |null|20220101| 60000|null|   123|
|  117|scoot7|       |null|20220101| 70000|null|   123|
|  118|scoot8|       |null|20220101| 80000|null|   123|
|  119|scoot9|       |null|20220101| 90000|null|   123|
|  222| cykim|       |null|20220101|100000|null|   123|
+-----+------+-------+----+--------+------+----+------+