We are using cloudera 5.7.0 with java 1.8.0_74 and we have spark 1.6.0, janusgraph 0.1.1, hbase 1.2.0.

I managed to bulkload 130GB of data into 1024 region hbase table in 2 hours 30 minute with 1024 spark executors (1-core,20gb memory). Each stage of blvp is configured to run 10240 tasks:

readGraph.compute(SparkGraphComputer).workers(10240).program(blvp).submit().get().

However I am unable to distribute the hbase data evenly across regions, they are pretty imbalanced. I suspect it is related to the conf value of ids.num-partitions.

Here is how I set the conf:

conf/my-janusgraph-hbase.properties:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=hbase

storage.batch-loading=true

storage.hbase.region-count=1024

ids.block-size=2000000

ids.renew-timeout=3600000

storage.buffer-size=10000

ids.num-partitions=1024

ids.partition=true

storage.hbase.table=myjanus

storage.hostname=x.x.x.x

cache.db-cache=true

cache.db-cache-clean-wait=20

cache.db-cache-time=180000

cache.db-cache-size=0.5

I even tried setting ids.num-partitions=10240; however the problem was not solved.

Should I still increase the ids.num-partitions value to an even higher value like 102400?

What is the difference cluster.max-partitions and ids.num-partitions. Is my problem related to cluster.max-partitions? Should I use it?

As far as I know ids.num-partitions value determines the number of randomly gathered prefixes that will be used in assigning ids to elements. And I read somewhere that setting ids.num-partitions to 10 times of region count will be enough; however it seems like that is not the case. And I do not want to increase ids.num-partitions further. Since I could not find any document related to the internals of cluster.max-partitions, I am really ignorant about it and need some help.

Thanks in advance,

Best,

Ali

Hi Ali,

I have never tried to optimize this myself, but maybe you should also look into the docs at

12.3.30. storage.hbase

...

storage.hbase.region-count

	The number of initial regions set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE
storage.hbase.regions-per-server	The number of regions per regionserver to set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE

Normally, HBase does not want many regions, but the number of regions times hdfs replication factor should be at least the number of active datanodes for maximum performance. I think some unbalance is inevitable as yarn will schedule executors unevenly and each executor will try to have local data access.

Further, you can look into HBase's region load balancer configuration, which enables HBase to move regions automatically.

HTH,    Marc

Op donderdag 15 juni 2017 16:04:28 UTC+2 schreef Ali ÖZER:

We are using cloudera 5.7.0 with java 1.8.0_74 and we have spark 1.6.0, janusgraph 0.1.1, hbase 1.2.0.

I managed to bulkload 130GB of data into 1024 region hbase table in 2 hours 30 minute with 1024 spark executors (1-core,20gb memory). Each stage of blvp is configured to run 10240 tasks:

readGraph.compute(SparkGraphComputer).workers(10240).program(blvp).submit().get().

However I am unable to distribute the hbase data evenly across regions, they are pretty imbalanced. I suspect it is related to the conf value of ids.num-partitions.

Here is how I set the conf:

conf/my-janusgraph-hbase.properties:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=hbase

storage.batch-loading=true

storage.hbase.region-count=1024

ids.block-size=2000000

ids.renew-timeout=3600000

storage.buffer-size=10000

ids.num-partitions=1024

ids.partition=true

storage.hbase.table=myjanus

storage.hostname=x.x.x.x

cache.db-cache=true

cache.db-cache-clean-wait=20

cache.db-cache-time=180000

cache.db-cache-size=0.5

I even tried setting ids.num-partitions=10240; however the problem was not solved.

Should I still increase the ids.num-partitions value to an even higher value like 102400?

What is the difference cluster.max-partitions and ids.num-partitions. Is my problem related to cluster.max-partitions? Should I use it?

As far as I know ids.num-partitions value determines the number of randomly gathered prefixes that will be used in assigning ids to elements. And I read somewhere that setting ids.num-partitions to 10 times of region count will be enough; however it seems like that is not the case. And I do not want to increase ids.num-partitions further. Since I could not find any document related to the internals of cluster.max-partitions, I am really ignorant about it and need some help.

Thanks in advance,

Best,

Ali

Hi Marc,

As far as I know, even if yarn schedules executors unevenly, it does not mean that the data written across hbase will be uneven.

The data is written hbase according to the key of the datum and the key-ranges of the regions, it does nothing to do with the node that the writer jvm is working on.

My executors are working on %90 of my nodes (it is not that uneven) however percentage of my empty regions is %90(900 of 1024 regions). If you were right the latter percentage would be %10 instead of %90.

If there is some other mechanism while assigning ids in distributed fashion, may you please keep me updated and elaborate on the mechanism.

Best,

Ali

15 Haziran 2017 Perşembe 22:51:14 UTC+3 tarihinde HadoopMarc yazdı:

Hi Ali,

I have never tried to optimize this myself, but maybe you should also look into the docs at

12.3.30. storage.hbase

...

storage.hbase.region-count

	The number of initial regions set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE
storage.hbase.regions-per-server	The number of regions per regionserver to set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE

Normally, HBase does not want many regions, but the number of regions times hdfs replication factor should be at least the number of active datanodes for maximum performance. I think some unbalance is inevitable as yarn will schedule executors unevenly and each executor will try to have local data access.

Further, you can look into HBase's region load balancer configuration, which enables HBase to move regions automatically.

HTH,    Marc

Op donderdag 15 juni 2017 16:04:28 UTC+2 schreef Ali ÖZER:

We are using cloudera 5.7.0 with java 1.8.0_74 and we have spark 1.6.0, janusgraph 0.1.1, hbase 1.2.0.

I managed to bulkload 130GB of data into 1024 region hbase table in 2 hours 30 minute with 1024 spark executors (1-core,20gb memory). Each stage of blvp is configured to run 10240 tasks:

readGraph.compute(SparkGraphComputer).workers(10240).program(blvp).submit().get().

However I am unable to distribute the hbase data evenly across regions, they are pretty imbalanced. I suspect it is related to the conf value of ids.num-partitions.

Here is how I set the conf:

conf/my-janusgraph-hbase.properties:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=hbase

storage.batch-loading=true

storage.hbase.region-count=1024

ids.block-size=2000000

ids.renew-timeout=3600000

storage.buffer-size=10000

ids.num-partitions=1024

ids.partition=true

storage.hbase.table=myjanus

storage.hostname=x.x.x.x

cache.db-cache=true

cache.db-cache-clean-wait=20

cache.db-cache-time=180000

cache.db-cache-size=0.5

I even tried setting ids.num-partitions=10240; however the problem was not solved.

Should I still increase the ids.num-partitions value to an even higher value like 102400?

What is the difference cluster.max-partitions and ids.num-partitions. Is my problem related to cluster.max-partitions? Should I use it?

As far as I know ids.num-partitions value determines the number of randomly gathered prefixes that will be used in assigning ids to elements. And I read somewhere that setting ids.num-partitions to 10 times of region count will be enough; however it seems like that is not the case. And I do not want to increase ids.num-partitions further. Since I could not find any document related to the internals of cluster.max-partitions, I am really ignorant about it and need some help.

Thanks in advance,

Best,

Ali

Hi Ali,

OK, I overlooked your config line "storage.hbase.region-count=1024". This is far too large a number, since HBase likes regions with a size of the order of 10GB, rather than the 130MB you requested. This 10 GB region split is probably a HBase global setting. It could be a similar number in your HBase cluster, so HBase just ignores the superfluous regions, unless you would configure the region split to a lower number manually (not advised).

Other comments from HBase experts are welcomed (I do not consider myself one).

Cheers,    Marc

Op vrijdag 16 juni 2017 09:57:12 UTC+2 schreef Ali ÖZER:

Hi Marc,

As far as I know, even if yarn schedules executors unevenly, it does not mean that the data written across hbase will be uneven.

The data is written hbase according to the key of the datum and the key-ranges of the regions, it does nothing to do with the node that the writer jvm is working on.

My executors are working on %90 of my nodes (it is not that uneven) however percentage of my empty regions is %90(900 of 1024 regions). If you were right the latter percentage would be %10 instead of %90.

If there is some other mechanism while assigning ids in distributed fashion, may you please keep me updated and elaborate on the mechanism.

Best,

Ali

15 Haziran 2017 Perşembe 22:51:14 UTC+3 tarihinde HadoopMarc yazdı:

Hi Ali,

I have never tried to optimize this myself, but maybe you should also look into the docs at

12.3.30. storage.hbase

...

storage.hbase.region-count

	The number of initial regions set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE
storage.hbase.regions-per-server	The number of regions per regionserver to set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE

Normally, HBase does not want many regions, but the number of regions times hdfs replication factor should be at least the number of active datanodes for maximum performance. I think some unbalance is inevitable as yarn will schedule executors unevenly and each executor will try to have local data access.

Further, you can look into HBase's region load balancer configuration, which enables HBase to move regions automatically.

HTH,    Marc

Op donderdag 15 juni 2017 16:04:28 UTC+2 schreef Ali ÖZER:

We are using cloudera 5.7.0 with java 1.8.0_74 and we have spark 1.6.0, janusgraph 0.1.1, hbase 1.2.0.

I managed to bulkload 130GB of data into 1024 region hbase table in 2 hours 30 minute with 1024 spark executors (1-core,20gb memory). Each stage of blvp is configured to run 10240 tasks:

readGraph.compute(SparkGraphComputer).workers(10240).program(blvp).submit().get().

However I am unable to distribute the hbase data evenly across regions, they are pretty imbalanced. I suspect it is related to the conf value of ids.num-partitions.

Here is how I set the conf:

conf/my-janusgraph-hbase.properties:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=hbase

storage.batch-loading=true

storage.hbase.region-count=1024

ids.block-size=2000000

ids.renew-timeout=3600000

storage.buffer-size=10000

ids.num-partitions=1024

ids.partition=true

storage.hbase.table=myjanus

storage.hostname=x.x.x.x

cache.db-cache=true

cache.db-cache-clean-wait=20

cache.db-cache-time=180000

cache.db-cache-size=0.5

I even tried setting ids.num-partitions=10240; however the problem was not solved.

Should I still increase the ids.num-partitions value to an even higher value like 102400?

What is the difference cluster.max-partitions and ids.num-partitions. Is my problem related to cluster.max-partitions? Should I use it?

As far as I know ids.num-partitions value determines the number of randomly gathered prefixes that will be used in assigning ids to elements. And I read somewhere that setting ids.num-partitions to 10 times of region count will be enough; however it seems like that is not the case. And I do not want to increase ids.num-partitions further. Since I could not find any document related to the internals of cluster.max-partitions, I am really ignorant about it and need some help.

Thanks in advance,

Best,

Ali

Hi Marco,

I think it has nothing to do with the region-count and hbase does not ignore any region in any circumstance. Since my regions are imbalaced (only 100 regions have data in them), data size per region is not outputsize/1024 it is outputsize/100.

My output size is not equal to my input size; it is 330GB therefore data size per region is not 130 MB it is 3 GB. Nevertheless as I said, I do not think that the problem is with the region count because I have managed to increase the balance of my regions by doing the followings:

I learned that the whole thing is about cluster.max-partitions parameter. Its default value is 32, I set it to 256 and changed nothing else and re-run the bulkloadervertexprogram and realized that non-empty region count was increased from 100 to 256. (when the parameter was 32; blvp loaded data into only 32 regions however hbase automatically splitted the oversized regions and the number of non-empty regions became 100). Therefore I realized that in order to fill 1024 regions of my 1024 regions I need to set the cluster.max-partitions to 1024. 

However there is one problem, when I increased the value of cluster.max-partitions from 32 to 256; the run-time of my bulkloadervertexprogram increased 5-times. I was able to load the whole data in almost 2 hours; now it is almost 10 hours. I think it is because each spark executor is trying to write 1024 region all at once. And I have 1024 spark executors; this means a lot of network 1024*1024. 

Due to the fact that I do not know the internals of the blvp and janus id assignment; I am not one hundred percent sure about all these.

Is there somebody knows the internals of janus, I would really appreciate that and I am pretty sure that this knowledge will really help me to solve the problem.

Best,

Ali

16 Haziran 2017 Cuma 14:55:00 UTC+3 tarihinde mar...@... yazdı:

Hi Ali,

OK, I overlooked your config line "storage.hbase.region-count=1024". This is far too large a number, since HBase likes regions with a size of the order of 10GB, rather than the 130MB you requested. This 10 GB region split is probably a HBase global setting. It could be a similar number in your HBase cluster, so HBase just ignores the superfluous regions, unless you would configure the region split to a lower number manually (not advised).

Other comments from HBase experts are welcomed (I do not consider myself one).

Cheers,    Marc

Op vrijdag 16 juni 2017 09:57:12 UTC+2 schreef Ali ÖZER:

Hi Marc,

As far as I know, even if yarn schedules executors unevenly, it does not mean that the data written across hbase will be uneven.

The data is written hbase according to the key of the datum and the key-ranges of the regions, it does nothing to do with the node that the writer jvm is working on.

My executors are working on %90 of my nodes (it is not that uneven) however percentage of my empty regions is %90(900 of 1024 regions). If you were right the latter percentage would be %10 instead of %90.

If there is some other mechanism while assigning ids in distributed fashion, may you please keep me updated and elaborate on the mechanism.

Best,

Ali

15 Haziran 2017 Perşembe 22:51:14 UTC+3 tarihinde HadoopMarc yazdı:

Hi Ali,

I have never tried to optimize this myself, but maybe you should also look into the docs at

12.3.30. storage.hbase

...

storage.hbase.region-count

	The number of initial regions set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE
storage.hbase.regions-per-server	The number of regions per regionserver to set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE

Normally, HBase does not want many regions, but the number of regions times hdfs replication factor should be at least the number of active datanodes for maximum performance. I think some unbalance is inevitable as yarn will schedule executors unevenly and each executor will try to have local data access.

Further, you can look into HBase's region load balancer configuration, which enables HBase to move regions automatically.

HTH,    Marc

Op donderdag 15 juni 2017 16:04:28 UTC+2 schreef Ali ÖZER:

We are using cloudera 5.7.0 with java 1.8.0_74 and we have spark 1.6.0, janusgraph 0.1.1, hbase 1.2.0.

I managed to bulkload 130GB of data into 1024 region hbase table in 2 hours 30 minute with 1024 spark executors (1-core,20gb memory). Each stage of blvp is configured to run 10240 tasks:

readGraph.compute(SparkGraphComputer).workers(10240).program(blvp).submit().get().

However I am unable to distribute the hbase data evenly across regions, they are pretty imbalanced. I suspect it is related to the conf value of ids.num-partitions.

Here is how I set the conf:

conf/my-janusgraph-hbase.properties:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=hbase

storage.batch-loading=true

storage.hbase.region-count=1024

ids.block-size=2000000

ids.renew-timeout=3600000

storage.buffer-size=10000

ids.num-partitions=1024

ids.partition=true

storage.hbase.table=myjanus

storage.hostname=x.x.x.x

cache.db-cache=true

cache.db-cache-clean-wait=20

cache.db-cache-time=180000

cache.db-cache-size=0.5

I even tried setting ids.num-partitions=10240; however the problem was not solved.

Should I still increase the ids.num-partitions value to an even higher value like 102400?

What is the difference cluster.max-partitions and ids.num-partitions. Is my problem related to cluster.max-partitions? Should I use it?

As far as I know ids.num-partitions value determines the number of randomly gathered prefixes that will be used in assigning ids to elements. And I read somewhere that setting ids.num-partitions to 10 times of region count will be enough; however it seems like that is not the case. And I do not want to increase ids.num-partitions further. Since I could not find any document related to the internals of cluster.max-partitions, I am really ignorant about it and need some help.

Thanks in advance,

Best,

Ali

Hi Ali,

Thanks for returning feedback about your experiments.

The docs have a section on graph partitioning, warning about a too large number of partitions and suggesting to use explicit partitioning if performance counts. They also suggest to take twice the number of region server instances as the region count.

http://docs.janusgraph.org/latest/graph-partitioning.html

HTH,   Marc

Op vrijdag 16 juni 2017 16:42:55 UTC+2 schreef Ali ÖZER:

Hi Marco,

I think it has nothing to do with the region-count and hbase does not ignore any region in any circumstance. Since my regions are imbalaced (only 100 regions have data in them), data size per region is not outputsize/1024 it is outputsize/100.

My output size is not equal to my input size; it is 330GB therefore data size per region is not 130 MB it is 3 GB. Nevertheless as I said, I do not think that the problem is with the region count because I have managed to increase the balance of my regions by doing the followings:

I learned that the whole thing is about cluster.max-partitions parameter. Its default value is 32, I set it to 256 and changed nothing else and re-run the bulkloadervertexprogram and realized that non-empty region count was increased from 100 to 256. (when the parameter was 32; blvp loaded data into only 32 regions however hbase automatically splitted the oversized regions and the number of non-empty regions became 100). Therefore I realized that in order to fill 1024 regions of my 1024 regions I need to set the cluster.max-partitions to 1024. 

However there is one problem, when I increased the value of cluster.max-partitions from 32 to 256; the run-time of my bulkloadervertexprogram increased 5-times. I was able to load the whole data in almost 2 hours; now it is almost 10 hours. I think it is because each spark executor is trying to write 1024 region all at once. And I have 1024 spark executors; this means a lot of network 1024*1024. 

Due to the fact that I do not know the internals of the blvp and janus id assignment; I am not one hundred percent sure about all these.

Is there somebody knows the internals of janus, I would really appreciate that and I am pretty sure that this knowledge will really help me to solve the problem.

Best,

Ali

16 Haziran 2017 Cuma 14:55:00 UTC+3 tarihinde mar...@... yazdı:

Hi Ali,

OK, I overlooked your config line "storage.hbase.region-count=1024". This is far too large a number, since HBase likes regions with a size of the order of 10GB, rather than the 130MB you requested. This 10 GB region split is probably a HBase global setting. It could be a similar number in your HBase cluster, so HBase just ignores the superfluous regions, unless you would configure the region split to a lower number manually (not advised).

Other comments from HBase experts are welcomed (I do not consider myself one).

Cheers,    Marc

Op vrijdag 16 juni 2017 09:57:12 UTC+2 schreef Ali ÖZER:

Hi Marc,

As far as I know, even if yarn schedules executors unevenly, it does not mean that the data written across hbase will be uneven.

The data is written hbase according to the key of the datum and the key-ranges of the regions, it does nothing to do with the node that the writer jvm is working on.

My executors are working on %90 of my nodes (it is not that uneven) however percentage of my empty regions is %90(900 of 1024 regions). If you were right the latter percentage would be %10 instead of %90.

If there is some other mechanism while assigning ids in distributed fashion, may you please keep me updated and elaborate on the mechanism.

Best,

Ali

15 Haziran 2017 Perşembe 22:51:14 UTC+3 tarihinde HadoopMarc yazdı:

Hi Ali,

I have never tried to optimize this myself, but maybe you should also look into the docs at

12.3.30. storage.hbase

...

storage.hbase.region-count

	The number of initial regions set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE
storage.hbase.regions-per-server	The number of regions per regionserver to set when creating JanusGraph’s HBase table	Integer	(no default value)	MASKABLE

Normally, HBase does not want many regions, but the number of regions times hdfs replication factor should be at least the number of active datanodes for maximum performance. I think some unbalance is inevitable as yarn will schedule executors unevenly and each executor will try to have local data access.

Further, you can look into HBase's region load balancer configuration, which enables HBase to move regions automatically.

HTH,    Marc

Op donderdag 15 juni 2017 16:04:28 UTC+2 schreef Ali ÖZER:

We are using cloudera 5.7.0 with java 1.8.0_74 and we have spark 1.6.0, janusgraph 0.1.1, hbase 1.2.0.

I managed to bulkload 130GB of data into 1024 region hbase table in 2 hours 30 minute with 1024 spark executors (1-core,20gb memory). Each stage of blvp is configured to run 10240 tasks:

readGraph.compute(SparkGraphComputer).workers(10240).program(blvp).submit().get().

However I am unable to distribute the hbase data evenly across regions, they are pretty imbalanced. I suspect it is related to the conf value of ids.num-partitions.

Here is how I set the conf:

conf/my-janusgraph-hbase.properties:

gremlin.graph=org.janusgraph.core.JanusGraphFactory

storage.backend=hbase

storage.batch-loading=true

storage.hbase.region-count=1024

ids.block-size=2000000

ids.renew-timeout=3600000

storage.buffer-size=10000

ids.num-partitions=1024

ids.partition=true

storage.hbase.table=myjanus

storage.hostname=x.x.x.x

cache.db-cache=true

cache.db-cache-clean-wait=20

cache.db-cache-time=180000

cache.db-cache-size=0.5

I even tried setting ids.num-partitions=10240; however the problem was not solved.

Should I still increase the ids.num-partitions value to an even higher value like 102400?

What is the difference cluster.max-partitions and ids.num-partitions. Is my problem related to cluster.max-partitions? Should I use it?

As far as I know ids.num-partitions value determines the number of randomly gathered prefixes that will be used in assigning ids to elements. And I read somewhere that setting ids.num-partitions to 10 times of region count will be enough; however it seems like that is not the case. And I do not want to increase ids.num-partitions further. Since I could not find any document related to the internals of cluster.max-partitions, I am really ignorant about it and need some help.

Thanks in advance,

Best,

Ali