hive输出格式转化，本例以json为例

本文主要是介绍hive输出格式转化，本例以json为例，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

 

hive默认是以tab为分隔符，分隔各个输出字段，如

1. hive> select get_json_object(json.value,'$.hour'),get_json_object(json.value,"$.channel") from json limit 10;

2. Total MapReduce jobs = 1

3. Launching Job 1 out of 1

4. Number of reduce tasks is set to 0 since there's no reduce operator

5. Starting Job = job_201304271626_0032, Tracking URL = http://linjianke:50030/jobdetails.jsp?jobid=job_201304271626_0032

6. Kill Command = /usr/lib/hadoop/bin/hadoop job -Dmapred.job.tracker=192.168.10.44:8021 -kill job_201304271626_0032

7. 2013-05-15 10:07:11,102 Stage-1 map = 0%, reduce = 0%

8. 2013-05-15 10:07:15,116 Stage-1 map = 100%, reduce = 0%

9. 2013-05-15 10:07:17,125 Stage-1 map = 100%, reduce = 100%

10. Ended Job = job_201304271626_0032

11. OK

12. 2013-04-07 16 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

13. 2013-04-07 22 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

14. 2013-04-08 00 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

15. 2013-04-08 01 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

16. 2013-04-09 01 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

17. 2013-04-09 07 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

18. 2013-04-09 16 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

19. 2013-04-09 17 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

20. 2013-04-09 21 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

21. 2013-04-10 01 f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8

22. Time taken: 9.531 seconds

 

因为业务需要，蒋输出转为json格式，可以用hive提供的transform函数

 

hive> select transform(get_json_object(json.value,'$.hour'),get_json_object(json.value,"$.channel")) using '/usr/bin/python transform.py aa bb' as (result string) from json limit 10;

 输出为

1. {"aa":"2013-04-07 16","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

2. {"aa":"2013-04-07 22","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

3. {"aa":"2013-04-08 00","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

4. {"aa":"2013-04-08 01","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

5. {"aa":"2013-04-09 01","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

6. {"aa":"2013-04-09 07","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

7. {"aa":"2013-04-09 16","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

8. {"aa":"2013-04-09 17","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

9. {"aa":"2013-04-09 21","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

10. {"aa":"2013-04-10 01","bb":"f32d98f8-8ac5-11e2-8a47-bc305bf4bcb8"}

 python脚本：对python还不大熟悉，写的可能比较搓

1.  

2. #!/usr/bin/python

3. #coding:utf8

4. import sys

5. for line in sys.stdin:

6. if len(line) == 0:

7. #print 'end'

8. break

9. if line.count('\n') == len(line):

10. #print 'continue'

11. continue

12. line=line.strip('\n')

13. arr=line.split('\t')

14. if len(arr) != len(sys.argv) - 1:

15. print 'error'

16. break

17. for i in range(0,len(arr)):

18. arr[i]='"%s":"%s"' % (sys.argv[i+1],arr[i])

19. content = ','.join(arr)

20. print '{%s}' % content

这篇关于hive输出格式转化，本例以json为例的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---