tag:blogger.com,1999:blog-8899645800948009496.post7879014102783549155..comments2024-03-18T15:34:34.751-07:00Comments on DBMS Musings: Hadoop's tremendous inefficiency on graph data management (and how to avoid it)Daniel Abadihttp://www.blogger.com/profile/16753133043157018521noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-8899645800948009496.post-54535934154092259832014-02-02T06:06:53.275-08:002014-02-02T06:06:53.275-08:00where do we get a code for the triple extractor fr...where do we get a code for the triple extractor from OWL file (similar to tabular form)Anonymoushttps://www.blogger.com/profile/15508048804710823086noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-27670131248233205072013-03-28T02:49:07.440-07:002013-03-28T02:49:07.440-07:00Totally agree. This is where things are made so co...Totally agree. This is where things are made so confusing. Why we need hadoop for graph processing. Shashank Jainhttps://www.blogger.com/profile/06436093643591683534noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-71425348211588098912011-08-28T16:18:36.280-07:002011-08-28T16:18:36.280-07:00Just shows that no matter what you use, schema des...Just shows that no matter what you use, schema design and choice still "matters" - even if you think you don't have a schema :)Greg Kemnitzhttps://www.blogger.com/profile/09831342731638036787noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-59923936932653114732011-08-10T07:37:17.709-07:002011-08-10T07:37:17.709-07:00I am really interested in your latest publication ...I am really interested in your latest publication as referenced in this blog. However the URL of http://cs-www.cs.yale.edu/homes/dna/papers/sw-graph-scale.pdf seems to be broken.<br />Could you please fix that?GWThttps://www.blogger.com/profile/11537637237847155273noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-74149257357716957082011-08-01T10:42:50.591-07:002011-08-01T10:42:50.591-07:00Very nice post, I really appreciate people sharing...Very nice post, I really appreciate people sharing knowledge.<br /><br />I have been using Hadoop for a season and recently I have to manage graph datasets. I agree with previous comments that Hadoop is amazing for its purpose, but it is not the solution for every problem. It reminds me a book (http://www.umiacs.umd.edu/~jimmylin/MapReduce-book-final.pdf) by Jimmy Lin explaining how to adapt MapReduce to graph problems. Why Hadoop instead well oriented solutions as Neo4J?<br /><br />However, your improvement seems really impresive and thank you again for writing this posts ;)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-64714160115145743512011-07-28T11:49:42.418-07:002011-07-28T11:49:42.418-07:00Daniel,
I did not read your paper yet but can ans...Daniel,<br /><br />I did not read your paper yet but can answer all three questions:<br /><br />1. Partitioner is plugable in Hadoop. You can create your own.<br />2. Replica placement is plugable as well (not sure its in the 0.21(2) but its definitely in the trunk )<br />3. HDFS provides low-level generic storage abstraction and you are can build any graph - optimized solution on top of HDFS API.Vladimir Rodionovhttps://www.blogger.com/profile/18159080975174540442noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-24941261290243242252011-07-23T12:08:38.502-07:002011-07-23T12:08:38.502-07:00Thanks for the post! Educational. I certainly agre...Thanks for the post! Educational. I certainly agree that one needs to pay close attention if your primary workload is running very inefficiently. Of course if a work load is using a small fraction of your resource, runs infrequently or is experimental, etc., then worrying about a factor of 2 or 10 might be premature optimization. I've found that the key is exposing your users to the economics of the resources they are using in multi-teneted environments.<br /><br />It will be interesting to see what can be done using the next version of Hadoop 0.23. That will support using custom execution frameworks within hadoop clusters. If your graph fits in RAM this should allow very optimal solutions. If not, it seems like it could still take advantage of the sorts of tricks you describe and those that HBase region servers use to do very well.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-72300219448212517122011-07-22T10:38:45.764-07:002011-07-22T10:38:45.764-07:00Graph partition is key to design graph algorithms ...Graph partition is key to design graph algorithms on Hadoop.xufei wanghttps://www.blogger.com/profile/02959710338374634129noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-64694250869048803612011-07-21T14:01:44.607-07:002011-07-21T14:01:44.607-07:00Hi Daniel, we came to the same conclusion in our w...Hi Daniel, we came to the same conclusion in our work on parallel abstractions for machine learning. As a consequence we developed an alternative computational abstraction, called GraphLab, to represent algorithms on graph structured data. By changing both the implementation and computational model we were able to both improve the running time and even the theoretical performance of our learning algorithms. You can find out more about GraphLab at http://graphlab.org.Joseph Gonzalezhttps://www.blogger.com/profile/12509128499280088966noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-34311910171406227452011-07-20T18:11:10.695-07:002011-07-20T18:11:10.695-07:00"Golden Orb" (http://www.goldenorbos.org..."Golden Orb" (http://www.goldenorbos.org/) is another open-source implementation of Pregel.. You may want to benchmark against that.Prafulla Wanihttps://www.blogger.com/profile/16339876403450744106noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-14689382613029345472011-07-20T15:17:19.442-07:002011-07-20T15:17:19.442-07:00Hi Daniel, Very nice work! and good point about th...Hi Daniel, Very nice work! and good point about the characteristics of the benchmark data and queries. Not sure if you have seen this SIGMOD'11 paper on RDF benchmarks, they have some interesting observations there: http://dx.doi.org/10.1145/1989323.1989340Oktie H.https://www.blogger.com/profile/01554245746131681648noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-6989649311742239682011-07-20T14:15:04.090-07:002011-07-20T14:15:04.090-07:00Mapper/reducer is meant for key/value-like process...Mapper/reducer is meant for key/value-like processing, not for graph and network data processing. If you are using Hadoop for network data processing, you are barking up the wrong tree. For graph processing, try neo4j, allegrograph or Google's pregel. <br /><br />Hope it helps.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-91931867171969312342011-07-20T10:16:15.568-07:002011-07-20T10:16:15.568-07:00I'm not familiar with any, but hopefully someo...I'm not familiar with any, but hopefully someone else can respond on this comment thread ...Daniel Abadihttps://www.blogger.com/profile/16753133043157018521noreply@blogger.comtag:blogger.com,1999:blog-8899645800948009496.post-43466475142910371962011-07-20T10:13:15.959-07:002011-07-20T10:13:15.959-07:00Hi Daniel, you mentioned in the post that neither ...Hi Daniel, you mentioned in the post that neither of HDFS and HBase data stores are optimized for graph data. I'm looking at the problem of using HBase to graph mining. Is there any paper about measuring HBase performance for graph data? Thanks.Vinh Ngoc Khuchttps://www.blogger.com/profile/10060144409365498125noreply@blogger.com