{"id":565,"date":"2012-04-08T16:12:36","date_gmt":"2012-04-08T21:12:36","guid":{"rendered":"http:\/\/crschmidt.net\/blog\/?p=565"},"modified":"2012-04-08T16:12:36","modified_gmt":"2012-04-08T21:12:36","slug":"multiprocessing-is-cool","status":"publish","type":"post","link":"https:\/\/crschmidt.net\/blog\/archives\/565\/multiprocessing-is-cool\/","title":{"rendered":"multiprocessing is cool"},"content":{"rendered":"<p>So, in general, I&#8217;ve avoided multithreaded processing; it&#8217;s one of those things that historically has been tricky to get right, and I don&#8217;t typically have embarrassingly parallel problems. <\/p>\n<p>Today, however, I was parsing a set of 25,000 HTML files using BeatifulSoup, to pull out a small set of data (~500 bytes of JSON per HTML file). I briefly tried to simplify some of the code, but then realized that the lion&#8217;s share of the CPU time was being spent on the initial parse; there wasn&#8217;t going to be a way to clean up my code enough to make the script that much faster, no matter how good the rest of my code was.<\/p>\n<p>Enter multiprocessing. With a 5 line change to my Python code, I was able to move from one core to four. Instead of:<\/p>\n<pre>\ndef handle_place():\n     for i in glob.glob(\"beerplaces\/*\"):\n           # Do stuff with I\n     return data\n<\/pre>\n<p>I have:<\/p>\n<pre>from multiprocessing import Pool\ndef handle_place(filename):\n    # Do stuff with filename\nif __name__ == \"__main__\":\n    p = Pool(4)\n    data = p.map(handle_place, glob.glob(\"places\/*\"))\n<\/pre>\n<p>Once I made the change, I went from using one CPU fully to using all four &#8212; and instead of taking 25 minutes to generate my output, the total time was under 7.<\/p>\n<p>multiprocessing is cool.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>So, in general, I&#8217;ve avoided multithreaded processing; it&#8217;s one of those things that historically has been tricky to get right, and I don&#8217;t typically have embarrassingly parallel problems. Today, however, I was parsing a set of 25,000 HTML files using BeatifulSoup, to pull out a small set of data (~500 bytes of JSON per HTML [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-565","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/posts\/565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/comments?post=565"}],"version-history":[{"count":0,"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/posts\/565\/revisions"}],"wp:attachment":[{"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/media?parent=565"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/categories?post=565"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/crschmidt.net\/blog\/wp-json\/wp\/v2\/tags?post=565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}