Hello,

Is there any supported way (e.g. a class/API) for deserializing raw data model rows, i.e. to get from raw Bigtable bytes to Vertex/edge list objects (in Java)?

https://docs.janusgraph.org/advanced-topics/data-model/

We're on the Cloud Bigtable storage backend, and it has excellent support for bulk exporting Bigtable rows (e.g. to Parquet in GCS), but we're unclear how to deserialize the raw Bigtable row/cell bytes back into usable Vertex objects.  If we were to build support for something like this, would it be a candidate for contribution back into the project?  Or is it misunderstanding the intended API/usage path?

Any thoughts greatly appreciated.  Thank you!

- Elliot

Hi Vinayak,

Please study the as(), select(), project() and cap() steps from the TinkerPop ref docs. The arguments of project() do not reference the keys of side effects but rather introduce new keys for its output. The query I provided above was tested in the TinkerPop modern graph, so I repeat my suggestion to try that one first and show in what way it fails to provide sensible output.

Best wishes,    Marc

Hi All,

I am trying to create nodes in graph and while reading created node id i am getting below exception:

org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend at io.vavr.API$Match$Case0.apply(API.java:3174)

at io.vavr.API$Match.of(API.java:3137)

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$0(CQLKeyColumnValueStore.java:125)

at io.vavr.control.Try.getOrElseThrow(Try.java:671)

at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:292)

at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$3.call(KCVSConfiguration.java:177)

at org.janusgraph.diskstorage.configuration.backend.KCVSConfiguration$3.call(KCVSConfiguration.java:174)

at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:147)

at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:161)

at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)

... 26 more

Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout during read query at consistency QUORUM (3 responses were required but only 2 replica responded)

I am trying to update janus client to not use QUORUM but LOCAL_ONE.  Setting below property but its not working.

 

JanusGraphFactory.Builder configProps = JanusGraphFactory.build();      configProps.set("storage.cql.read-consistency-level", “LOCAL_ONE”);

 

In CQLTransaction class i see its getting set by reading value from CustomOption

public CQLTransaction(final BaseTransactionConfig config) {
    super(config);
    this.readConsistencyLevel = ConsistencyLevel.valueOf(getConfiguration().getCustomOption(READ_CONSISTENCY));
    this.writeConsistencyLevel = ConsistencyLevel.valueOf(getConfiguration().getCustomOption(WRITE_CONSISTENCY));
}

I did tied  seeing value by using customOption() method of TransactionBuilder but no luck.

 

GraphTraversalSource g = janusGraph.buildTransaction()
        .customOption("storage.cql.read-consistency-level","LOCAL_ONE")
        .start().traversal();

 

Could you please help me to fix it?


Thanks & Regards,
Anjani

 

Thank you Ted, I am also interested in watching this video on OLAP later, it is like 00:30 in Melbourne, not too easy to get up in the morning...
Regards to presenters, Christopher.

Hi Ted,

Hi Anjani,

The following section of the TinkerPop ref docs gives an example of how to reuse the output RDD of one job in a follow-up gremlin OLAP job.
https://tinkerpop.apache.org/docs/3.4.10/reference/#interacting-with-spark

Best wishes,   Marc

Hi Boxuan,

Yes, I did not finish my argument. What I tried to suggest: if the hadoop CLI command checks the GENERIC_OPTIONS env variable, then maybe also the mapreduce java client called by JanusGraph checks the GENERIC_OPTIONS env variable.

The (old) blog below suggests, however, that this behavior is not present by default but requires the janusgraph code to run hadoop's ToolRunner. So, just see if this is any better than what you had in mind to implement.
https://hadoopi.wordpress.com/2013/06/05/hadoop-implementing-the-tool-interface-for-mapreduce-driver/

Best wishes,    Marc

Hi Marc, you are right, we are indeed using this -files option :)

Hi Boxuan,

Using existing mechanisms for configuring mapreduce would be nicer, indeed.

Upon reading this hadoop command, I see a GENERIC_OPTIONS env variable read by the mapreduce client, that can have a -files option. Maybe it is possible to include a jaas file that points to the (already installed?) keytab file on the workers?

Best wishes,     Marc

We have been using a yarn cluster to run MapReduce reindexing (Cassandra + Elasticsearch) for a long time. Recently, we introduced Kerberos-based authentication to the Elasticsearch cluster, meaning that worker nodes need to authenticate via a keytab file.

We managed to achieve so by using a hadoop command to include the keytab file when submitting the MapReduce job. Hadoop automatically copies this file and distributes it to the working directory of all worker nodes. This works well for us, except that we have to make changes to MapReduceIndexManagement class so that it accepts an org.apache.hadoop.conf.Configuration object (which is created by org.apache.hadoop.util.ToolRunner) rather than instantiate one by itself. We are happy to submit a PR for this, but I would like to hear if there is any better way of handling this.

Cheers,
Boxuan

Hello,

I encounter the error describe in the issue #1916 (https://github.com/JanusGraph/janusgraph/issues/1916) on a mixed index (lucene). When I list property keys of the index, they are all duplicated.

I haven't identified the root cause and I don't know how to reproduce it.

I would like to find a solution to repair the inconsistency,  without loosing my data.

The exposed API of JanusGraphManagement doesn't seem to be helpful: mixed index can't be removed and the IllegalArgumentException is raised when I try to retrieve the index status. Removing the index in Lucene or unconfiguring index backend doesn't help either.

So I've tried to find a solution using internal API. Is it safe to delete Lucene data files and remove the schema vertex related to the mixed index ?

((ManagementSystem)mgmt).getWrappedTx()
  .getSchemaVertex(JanusGraphSchemaCategory.GRAPHINDEX.getSchemaName("indexName"))
  .remove()

Is there a reason to not permit to remove a mixed index ?

Best wishes

Toom.

On Thu, May 6, 2021 at 11:36 PM, <hadoopmarc@...> wrote:

Marc

Hi Marc,

Thanks for the reply. Yeah, it's true that a multi-node setup uses the same storage backend. However, I am referring to here is the Janusgraph database cache - https://docs.janusgraph.org/basics/cache/#database-level-caching Which is using Janusgraph heap memory.

Hi All,

I am using connected component vertex program to find all the connected nodes in graph and then using that RDD for further processing in graph. I want to store that RDD at some output location so that i can re-use the RDD and don't have to run connected component vertex program which is time consuming. 

I see in tinker-pop library we have OutputFormatRDD  to save data. I tired

outputFormatRDD.writeGraphRDD(graphComputerConfiguration, uniqueRDD);  ## connected but its throwing class cast exception as connected component vertex program output RDD value is a list which can not be cast to VertexWritable

 

 outputFormatRDD.writeMemoryRDD(graphComputerConfiguration, "memoryKey",  uniqueRDD);  ## Its saving RDD by creating memory key folder name at output location.

Not able to read RDD through InputFormatRDD.readMemoryRDD() as its looking for data files as per class SequenceFileInputFormat class. 

Am i missing any thing? Please let me know if you have tired some these methods? Want to check if we can use out of box methods before proceeding with our own?

Thanks,
Anjani

On Tue, May 11, 2021 at 04:56 PM, <hadoopmarc@...> wrote:

ids.num-partitions

Thanks for help Marc, i will try updating value of  ids.num-partitions = number of executors.

Hi Anjani,

It is a while ago I did this myself. I interpret ids.num-partitions as a stock of reserved id blocks that can be delegated to a janugraph instance. It does not have a large value to not waste ids space.

Actually, parallel tasks is not the number we want. We want the ids.num-partitions to be equal to the number of janusgraph instances, because initially all janusgraph instances ask for an ids block at the same time. Note that the cores in a spark executor can share the same janusgraph instance if you use a singleton object for that.

So, if you have 50 executors with 5 cores each (and using a singleton janusgraph instance), I would try ids.num-partitions =50

Best wishes,    Marc

We have 250 parallel spark task running for creating node/edges.
I didn't get parallel tasks -  (for setting ids.num-partitions)? Could you please help me on it?