get "java.lang.OutOfMemoryError: Java heap space" when running query Square and House on Friendster

[guowentian]

Hi, I run SEED in a cluster of 40 machines, each with 32GB memory. I find that when I run Square and House pattern on Friendster, some errors occur due to insufficient memory.
The following is a snapshot of the complaining errors.
At first time, I try to set the mapper memory and reducer memory to be 2048M and 4096M, while mapreduce.map.java.opts and mapreduce.reduce.java.opts are set to be 1638M and 3276M.
In the second time, I try to set the mapper memory and reducer memory to be 12G and 24G, while mapreduce.map.java.opts and mapreduce.reduce.java.opts are set to be 9G and 19G.
But I fail both times.
Any suggestion ??

Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:56)
at org.apache.hadoop.io.BoundedByteArrayOutputStream.(BoundedByteArrayOutputStream.java:46)
at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.(InMemoryMapOutput.java:63)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.unconditionalReserve(MergeManagerImpl.java:309)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.reserve(MergeManagerImpl.java:299)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:511)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:333)
at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:193)
.....

[lidongze0629]

@guowentian when I run subgenum_pre.py, I set the data: testdata , <ps: testdata is a file, each line just like 1,2 > so I create a dir in hdfs://user/username/testdata , but I don't know where to put my testdata ? I only create a dir named testdata in hdfs , have not put the data file to hdfs, could you tell the detail to run this script? thank you

[guowentian]

@lidongze0629 I think you should upload testdata into the corresponding directory in hadoop file system

[lidongze0629]

very thanks for your reply, I think you are right, but I don not which is the corresponding directory for my testdata , I have upload to /user/lidongze/testdata/testdata, /user/lidongze/testdata/adjList2.0/testdata /user/lidongze/testdata/dd/testdata, but it all looks like wrong...

[guowentian]

@lidongze0629 /user/lidongze/testdata/${your_data_file}

[lidongze0629]

thanks, I will try it now .