{"id":408,"date":"2013-06-18T04:26:19","date_gmt":"2013-06-18T04:26:19","guid":{"rendered":"https:\/\/shan.info\/new\/lexicographically-sorting-large-files-in-linux\/"},"modified":"2013-06-18T04:26:19","modified_gmt":"2013-06-18T04:26:19","slug":"lexicographically-sorting-large-files-in-linux","status":"publish","type":"post","link":"https:\/\/shan.info\/new\/lexicographically-sorting-large-files-in-linux\/","title":{"rendered":"Lexicographically sorting large files in Linux"},"content":{"rendered":"<p style=\"margin-top: 0px; margin-bottom: 10px; color: #333333; font-family: kepler-std, Helvetica, Arial, sans-serif; line-height: 25.1875px;\">When I hear the word \u201csort\u201d my first thought is usually \u201cHadoop\u201d! Yes, sorting is one thing that Hadoop does well, but if you\u2019re working with large files in Linux the built-in sort command is often all you need.<\/p>\n<p style=\"margin-top: 0px; margin-bottom: 10px; color: #333333; font-family: kepler-std, Helvetica, Arial, sans-serif; line-height: 25.1875px;\">Let\u2019s say you have a large file on a host with 2GB or more of main memory free. The following&nbsp;<a style=\"color: #0088cc;\" href=\"http:\/\/www.oreillynet.com\/linux\/cmd\/cmd.csp?path=s\/sort\">sort<\/a>&nbsp;command is a efficient way to&nbsp;<a style=\"color: #0088cc;\" href=\"http:\/\/en.wikipedia.org\/wiki\/Lexicographical_order\">lexicographically<\/a>-order large files.<\/p>\n<pre style=\"padding: 9.5px; font-family: Monaco, Menlo, Consolas, 'Courier New', monospace; font-size: 13px; color: #333333; margin: 0px 0px 10px; line-height: 20px; white-space: pre-wrap; background-color: #f5f5f5; border: 1px solid rgba(0, 0, 0, 0.14902);\"><code style=\"padding: 0px; font-family: Monaco, Menlo, Consolas, 'Courier New', monospace; font-size: 12px; color: inherit; background-color: transparent; border: 0px;\">LC_COLLATE=C sort --buffer-size=1G --temporary-directory=.\/tmp --unique bigfile.txt<\/code><\/pre>\n<p style=\"margin-top: 0px; margin-bottom: 10px; color: #333333; font-family: kepler-std, Helvetica, Arial, sans-serif; line-height: 25.1875px;\">Let\u2019s break this command down and examine each part in detail.<\/p>\n<p style=\"margin-top: 0px; margin-bottom: 10px; color: #333333; font-family: kepler-std, Helvetica, Arial, sans-serif; line-height: 25.1875px;\"><img decoding=\"async\" src=\"https:\/\/shan.info\/new\/wp-content\/uploads\/images\/sorting-large-files-linux.png\" alt=\"\" \/><\/p>\n","protected":false},"excerpt":{"rendered":"<p>When I hear the word \u201csort\u201d my first thought is usually \u201cHadoop\u201d! Yes, sorting is one thing that Hadoop does well, but if you\u2019re working with large files in Linux the built-in sort command is often all you need. Let\u2019s say you have a large file on a host with 2GB or more of main [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[33],"tags":[],"class_list":["post-408","post","type-post","status-publish","format-standard","hentry","category-centos-6"],"_links":{"self":[{"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/posts\/408","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/comments?post=408"}],"version-history":[{"count":0,"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/posts\/408\/revisions"}],"wp:attachment":[{"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/media?parent=408"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/categories?post=408"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/shan.info\/new\/wp-json\/wp\/v2\/tags?post=408"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}