海量数据按行数进行切分

本文主要是介绍海量数据按行数进行切分，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

import os
import codecsdef split_file(read_file,write_file):
    readfile = open(read_file,"r")j = 0
    for line in readfile:
        line = line.strip()line_str = line.split('\t')if j%10000 == 0:  #按10000条数据切分
            num = int(j/10000) #设置文件的命名规则从0开始
            _write_file = write_file+str(num)+".txt"
            if os.path.exists(_write_file):
                os.remove(_write_file)writefile = open(_write_file,"w")new_line = "'"+line_str[1]+"'"+","
        writefile.write(new_line+ "\n")j += 1

    readfile.close()writefile.close()if __name__=='__main__':
    read_file = "C:\\Users\\ziyht\\Desktop\\mysql_data_mac.txt"
    write_file = "C:\\Users\\ziyht\\Desktop\\test\\mac_"
    split_file(read_file,write_file)

这篇关于海量数据按行数进行切分的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---