Date   

[PROPOSAL] new repository for janusgraph-foundationdb

Jason Plurad <plu...@...>
 

Several developers from the community have expressed interest in collaborating on a storage backend for FoundationDB. As discussed on this thread, they each have a separate fork (no license changes) from the original experoinc/janusgraph-foundationdb repo that Ted Wilmes created (no updates in ~2 years).

I propose that we create a new project janusgraph-foundationdb under the JanusGraph org so that developers can work together to merge their forks and move forward.

Let's give it 72 hours for lazy consensus.

I'm +1 for this proposal.

-- Jason


Re: Janusgraph functionality issue

Dmitry Kovalev <dk.g...@...>
 

Hi Shiva,

Thanks for taking time to dive into the issue and deciding to contribute.

Please have a look at this page: https://github.com/JanusGraph/janusgraph/blob/master/CONTRIBUTING.md - it describes the steps to contribute, which are pretty standard.

Thanks,
Dmitry

On Thu, 11 Jun 2020 at 10:40, <shivain...@...> wrote:
Hello Everyone,

I have found the root cause for this issue and also have the fix for the same.

Please guide me if I can contribute for this project as I'm new to this community.

Thanks
Shiva

On Monday, June 8, 2020 at 6:50:04 PM UTC+5:30, Shiva Krishnan wrote:
Hi,

I have noticed a corruption in index after reindexing vertex-centric indices with direction 'IN'.

Issue : A self-link is getting added to every vertex after reindexing.

To demonstrate this, I have taken a very small graph consists of two vertices (A & B) and one edge connecting the two.
Edge Label -> link
Edge has one property key (Integer) -> assocKind

//Creating the Vertex Centrix Index
gremlin > edgeLabel = mgmt.getEdgeLabel("link");
gremlin > assocKind= mgmt.getPropertyKey("assocKind")
gremlin > mgmt.buildEdgeIndex(edgeLabel, "myVertexCentricIndex", Direction.IN, assocKind);

Please note i have given the direction IN for the index.


//Creating the Edge from A to B: (a and b are vertices)
a.addEdge("link",b,"assocKind",1)

Output Before Reindexing :
gremlin> g.V().has('name' , 'A').inE().hasLabel('link').has('assocKind',1)
//no IN edges to vertex A  Correct

gremlin> g.V().has('name' , 'A').outE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-1pqw][14488-link->80024]   Correct

Now I ran the reindex command
// 'index' is the vertex centric index which is created above
gremlin> m.updateIndex(index, SchemaAction.REINDEX).get()

Output After Reindexing :
gremlin> g.V().has('_objId','A').inE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-b6g][14488-link->14488] Wrong

gremlin> g.V().has('_objId','A').bothE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-1pqw][14488-link->80024]
==>e[4e1f-b6g-1bit-b6g][14488-link->14488] Unexpected Link

This issue happens in janusgraph 0.5.2 also.

Thanks
Shiva

--
You received this message because you are subscribed to the Google Groups "JanusGraph developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to janusgr...@....
To view this discussion on the web visit https://groups.google.com/d/msgid/janusgraph-dev/ea55b56a-dc8b-456e-b8c9-071de9b9f35ao%40googlegroups.com.


Re: Janusgraph functionality issue

shivain...@...
 

Hello Everyone,

I have found the root cause for this issue and also have the fix for the same.

Please guide me if I can contribute for this project as I'm new to this community.

Thanks
Shiva

On Monday, June 8, 2020 at 6:50:04 PM UTC+5:30, Shiva Krishnan wrote:
Hi,

I have noticed a corruption in index after reindexing vertex-centric indices with direction 'IN'.

Issue : A self-link is getting added to every vertex after reindexing.

To demonstrate this, I have taken a very small graph consists of two vertices (A & B) and one edge connecting the two.
Edge Label -> link
Edge has one property key (Integer) -> assocKind

//Creating the Vertex Centrix Index
gremlin > edgeLabel = mgmt.getEdgeLabel("link");
gremlin > assocKind= mgmt.getPropertyKey("assocKind")
gremlin > mgmt.buildEdgeIndex(edgeLabel, "myVertexCentricIndex", Direction.IN, assocKind);

Please note i have given the direction IN for the index.


//Creating the Edge from A to B: (a and b are vertices)
a.addEdge("link",b,"assocKind",1)

Output Before Reindexing :
gremlin> g.V().has('name' , 'A').inE().hasLabel('link').has('assocKind',1)
//no IN edges to vertex A  Correct

gremlin> g.V().has('name' , 'A').outE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-1pqw][14488-link->80024]   Correct

Now I ran the reindex command
// 'index' is the vertex centric index which is created above
gremlin> m.updateIndex(index, SchemaAction.REINDEX).get()

Output After Reindexing :
gremlin> g.V().has('_objId','A').inE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-b6g][14488-link->14488] Wrong

gremlin> g.V().has('_objId','A').bothE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-1pqw][14488-link->80024]
==>e[4e1f-b6g-1bit-b6g][14488-link->14488] Unexpected Link

This issue happens in janusgraph 0.5.2 also.

Thanks
Shiva


Re: Official support for FoundationDB?

f.gri...@...
 

I'm of course interested and would like to offer my support for everything that needs to be done.

-- Florian

rngcntr -- Sorry, I didn't see your first name... are you interested as well?


Re: Official support for FoundationDB?

Jason Plurad <plu...@...>
 

Sounds like there is some interest here from Chris (blindenvy) and Yuvaraj (skyrocknroll) on a common fork.

rngcntr -- Sorry, I didn't see your first name... are you interested as well?

I'd think the next steps would be to get a vote approved first, then we'd need a repo to seed to the project. There are a few options there on how to proceed with that, but let's get the above answered first before moving to a vote.

-- Jason

On Friday, June 5, 2020 at 9:51:01 PM UTC-4, Yuvaraj Loganathan wrote:
We would be really interested in a common fork.


Janusgraph functionality issue

shivain...@...
 

Hi,

I have noticed a corruption in index after reindexing vertex-centric indices with direction 'IN'.

Issue : A self-link is getting added to every vertex after reindexing.

To demonstrate this, I have taken a very small graph consists of two vertices (A & B) and one edge connecting the two.
Edge Label -> link
Edge has one property key (Integer) -> assocKind

//Creating the Vertex Centrix Index
gremlin > edgeLabel = mgmt.getEdgeLabel("link");
gremlin > assocKind= mgmt.getPropertyKey("assocKind")
gremlin > mgmt.buildEdgeIndex(edgeLabel, "myVertexCentricIndex", Direction.IN, assocKind);

Please note i have given the direction IN for the index.


//Creating the Edge from A to B: (a and b are vertices)
a.addEdge("link",b,"assocKind",1)

Output Before Reindexing :
gremlin> g.V().has('name' , 'A').inE().hasLabel('link').has('assocKind',1)
//no IN edges to vertex A  Correct

gremlin> g.V().has('name' , 'A').outE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-1pqw][14488-link->80024]   Correct

Now I ran the reindex command
// 'index' is the vertex centric index which is created above
gremlin> m.updateIndex(index, SchemaAction.REINDEX).get()

Output After Reindexing :
gremlin> g.V().has('_objId','A').inE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-b6g][14488-link->14488] Wrong

gremlin> g.V().has('_objId','A').bothE().hasLabel('link').has('assocKind',1)
==>e[4e1f-b6g-1bit-1pqw][14488-link->80024]
==>e[4e1f-b6g-1bit-b6g][14488-link->14488] Unexpected Link

This issue happens in janusgraph 0.5.2 also.

Thanks
Shiva


Edge Filter in Janusgraph when working with Spark

rafi ansari <rafi1...@...>
 

Hi All,

I am currently working on using Janusgraph in batch mode using Spark.

I am facing a problem on filtering the edges by label.

Below are the specifications:
Spark = 2.4.5
Janusgraph = 0.5.0

Below is the configuration file for Spark:

conf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph")
conf.setProperty("gremlin.hadoop.graphReader", "org.janusgraph.hadoop.formats.cql.CqlInputFormat")
conf.setProperty("gremlin.hadoop.graphWriter", "org.apache.hadoop.mapreduce.lib.output.NullOutputFormat")
conf.setProperty("spark.cassandra.connection.host", "127.0.0.1")
conf.setProperty("janusgraphmr.ioformat.conf.storage.backend", "cql")
conf.setProperty("janusgraphmr.ioformat.conf.storage.hostname", "127.0.0.1")
conf.setProperty("janusgraphmr.ioformat.conf.storage.port", 9042)
conf.setProperty("janusgraphmr.ioformat.conf.storage.cql.keyspace", "graph_db_1")
conf.setProperty("janusgraphmr.ioformat.conf.index.search.backend", "elasticsearch")
conf.setProperty("janusgraphmr.ioformat.conf.index.search.hostname", "127.0.0.1")
conf.setProperty("janusgraphmr.ioformat.conf.index.search.port", 9200)
conf.setProperty("janusgraphmr.ioformat.conf.index.search.index-name", "graph_1")
conf.setProperty("cassandra.input.partitioner.class","org.apache.cassandra.dht.Murmur3Partitioner")
conf.setProperty("cassandra.input.widerows",true)
conf.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.setProperty("spark.kryo.registrator", "org.janusgraph.hadoop.serialize.JanusGraphKryoRegistrator")

Below is the Spark code using newAPIHadoopRDD 


val hadoopConfiguration = ConfUtil.makeHadoopConfiguration(conf)

val rdd: RDD[(NullWritable, VertexWritable)] =spark.sparkContext.newAPIHadoopRDD(hadoopConfiguration,hadoopConfiguration.
getClass(Constants.GREMLIN_HADOOP_GRAPH_READER, classOf[InputFormat[NullWritable, VertexWritable]]).
asInstanceOf[Class[InputFormat[NullWritable, VertexWritable]]],classOf[NullWritable], classOf[VertexWritable])

The above lines give an RDD as output.

rdd: org.apache.spark.rdd.RDD[(org.apache.hadoop.io.NullWritable, org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable)]

rdd.map{case (x,y)=>y.asInstanceOf[VertexWritable]}

res17: Array[String] = Array(v[8344], v[12440], v[4336], v[4320], v[4136], v[8416], v[8192], v[4248], v[4344], v[8432], v[12528], v[4096])

From the res17 above, not sure how to filter the edges by labels


TIA

Regards

Rafi


Re: Official support for FoundationDB?

Yuvaraj Loganathan <uva...@...>
 

We would be really interested in a common fork.


Re: Official support for FoundationDB?

Jason Plurad <plu...@...>
 

+1. I support having a janusgraph-foundationdb repository under the main JanusGraph org on GitHub. There is precedent for this with janusgraph-docker, language-specific client bindings (janusgraph-dotnet and janusgraph-python), and janusgraph-ambari.

It comes down to whether there is a consensus desire to collaborate together instead of having separate forks. There would be some work from those fork authors to agree on how to merge those forks into the one that will serve as the initial repo. There would be more restrictions on getting code committed since the project would be expected to have a code review and approval process in place, like the other JanusGraph repos. Keep in mind that having a repo under the JanusGraph org doesn't mean that it needs to be production worthy from the start.

Thanks to both of you for getting the ball rolling here with FoundationDB. Happy to hear more feedback/discussion from the community.

-- Jason

On Saturday, May 30, 2020 at 5:57:13 PM UTC-4, Christopher Jackson wrote:
I agree we would have to clearly state all limitations the adapter has so everyone is aware of them. 

Would love to hear additional feedback from the community on the topic of making this adapter official. 

On Monday, May 25, 2020 at 3:21:22 AM UTC-4, f....@... wrote:
Hi Christopher,
thanks for offering your support!

I did indeed put a lot of effort into my fork (rngcntr), but I worked alone and rewrote a significant amount of code so it definitely lacks an in-depth review. Other than that, most of what I did was stress testing and finding bugs that need to be fixed. There are stil some open issues but all in all I'm quite happy with the current state of the storage adapter.

What still worries me is the effects of the transaction time limit of FDB. Officially supporting FDB as a storage backend would currently leave the user in charge of handling these problems. Retrying can not solve all problems as some queries will never make it through the 5s limit, even when tried over and over again. This results in the FDB adapter only supporting a subset of all possible queries and this should be stated clearly in case it is officially released.


Re: Official support for FoundationDB?

Christopher Jackson <jackson.ch...@...>
 

I agree we would have to clearly state all limitations the adapter has so everyone is aware of them. 

Would love to hear additional feedback from the community on the topic of making this adapter official. 


On Monday, May 25, 2020 at 3:21:22 AM UTC-4, f....@... wrote:
Hi Christopher,
thanks for offering your support!

I did indeed put a lot of effort into my fork (rngcntr), but I worked alone and rewrote a significant amount of code so it definitely lacks an in-depth review. Other than that, most of what I did was stress testing and finding bugs that need to be fixed. There are stil some open issues but all in all I'm quite happy with the current state of the storage adapter.

What still worries me is the effects of the transaction time limit of FDB. Officially supporting FDB as a storage backend would currently leave the user in charge of handling these problems. Retrying can not solve all problems as some queries will never make it through the 5s limit, even when tried over and over again. This results in the FDB adapter only supporting a subset of all possible queries and this should be stated clearly in case it is officially released.


Re: Official support for FoundationDB?

f.gri...@...
 

Hi Christopher,
thanks for offering your support!

I did indeed put a lot of effort into my fork (rngcntr), but I worked alone and rewrote a significant amount of code so it definitely lacks an in-depth review. Other than that, most of what I did was stress testing and finding bugs that need to be fixed. There are stil some open issues but all in all I'm quite happy with the current state of the storage adapter.

What still worries me is the effects of the transaction time limit of FDB. Officially supporting FDB as a storage backend would currently leave the user in charge of handling these problems. Retrying can not solve all problems as some queries will never make it through the 5s limit, even when tried over and over again. This results in the FDB adapter only supporting a subset of all possible queries and this should be stated clearly in case it is officially released.


Official support for FoundationDB?

jackson.ch...@...
 

Hi everyone,

Seems there may be a good deal of interest in having support for FoundationDB as a backend storage for Janusgraph. Ted Wilmes had created a storage adapter https://github.com/experoinc/janusgraph-foundationdb that hasn't seen any updates by Ted for some time, however other teams have picked up where Ted left off with a few forks seeing some activity:

https://github.com/rngcntr/janusgraph-foundationdb
https://github.com/skyrocknroll/janusgraph-foundationdb
https://github.com/blindenvy/janusgraph-foundationdb

I wanted to start a conversation about working towards getting this adapter added as part of the main janusgraph code base and making FDB an officially supported backend.

Personally, I am interested in seeing this happen as I work for IBM and we are planning on using Janusgraph on top of FoundationDB for a few projects. The https://github.com/blindenvy/janusgraph-foundationdb fork is where some of our team members have started to make changes. We originally forked from experoinc repo and only noticed the other forks after the fact. It seems both the rngcntr and skyrocknroll forks have seen a good deal of changes and may be better forks to choose as a starting point to bring in to the official code base.  

If the community decides they want to do this I would like to offer my assistance in helping bring the code in and adding documentation to jump start the process. 

I look forward to hearing everyones thoughts on this topic. 

Regards,
Christopher Jackson


Read or write in JanusGraph with Spark GraphFrames

rafi ansari <rafi1...@...>
 

Hi all,

Is there any documentation on how to connect Spark GraphFrame to JanusGraph for BULK read/write operations?

JanusGraph docs (https://docs.janusgraph.org/advanced-topics/hadoop/) shows how to do OLAP on Gremlin, but I could not find any documentation or source doing OLAP using Spark GraphFrame.

Thanks

Rafi


[RESULT][VOTE] JanusGraph 0.5.2 release

Oleksandr Porunov <alexand...@...>
 

This vote is now closed with a total of 3 +1s, no +0s and no -1s. The results are:

BINDING VOTES:

+1  (3 -- Oleksandr Porunov, Florian Hockmann, Jan Jansen)
0   (0)
-1  (0)

NON-BINDING VOTES:

+1 (0)
0  (0)
-1 (0)

Thank you very much,
Oleksandr Porunov


Re: [VOTE] JanusGraph 0.5.2 release

Jan Jansen <faro...@...>
 

I did a quick test of both binary distributions. VOTE +1

I think could work on releasing an extra distribution for hbase1 which is used for google bigtable, if I'm correct.


Re: [VOTE] JanusGraph 0.5.2 release

Florian Hockmann <f...@...>
 

I checked the release notes and did a quick test of both binary distributions. VOTE +1

Am Mittwoch, 6. Mai 2020 02:51:49 UTC+2 schrieb Oleksandr Porunov:

Hello,

We are happy to announce that JanusGraph 0.5.2 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.5.2

A full binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-full-0.5.2.zip

A truncated binary distribution is provided:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-0.5.2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-0.5.2-doc.zip

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.5.2

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.5/docs/changelog.md#version-052-release-date-may-3-2020

This [VOTE] will open for the next 3 days --- closing Saturday, May 9, 2020 at 12:50 AM GMT.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


[VOTE] JanusGraph 0.5.2 release

Oleksandr Porunov <alexand...@...>
 

Hello,

We are happy to announce that JanusGraph 0.5.2 is ready for release.

The release artifacts can be found at this location:
        https://github.com/JanusGraph/janusgraph/releases/tag/v0.5.2

A full binary distribution is provided for user convenience:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-full-0.5.2.zip

A truncated binary distribution is provided:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-0.5.2.zip

The GPG key used to sign the release artifacts is available at:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/KEYS

The docs can be found here:
        https://github.com/JanusGraph/janusgraph/releases/download/v0.5.2/janusgraph-0.5.2-doc.zip

The release tag in Git can be found here:
        https://github.com/JanusGraph/janusgraph/tree/v0.5.2

The release notes are available here:
        https://github.com/JanusGraph/janusgraph/blob/v0.5/docs/changelog.md#version-052-release-date-may-3-2020

This [VOTE] will open for the next 3 days --- closing Saturday, May 9, 2020 at 12:50 AM GMT.
All are welcome to review and vote on the release, but only votes from TSC members are binding.
My vote is +1.

Thank you,
Oleksandr Porunov


Janusgraph (Cassandra + E.S ) , OLAP BULK ingestion , issues with ES (SecondaryPersistence used to store indexes) commit and rollback funcationlaity

Ramesh Babu Y <ramesh...@...>
 

We are using jansugraph with Cassandra + ES combination , we are doing data ingestion with OLAP mode , since we are submitting batch requests , and we are using commit method to commit the transaction and their by to start actual data ingestion , here problem is first it will do cassandra commit and if their is any issue during commit janusgraph core classes are doing rolback and in same janus graph core classes , their is no rollback performed when their is issues during ES commit . below the janusgraph class where we identified the issue 

In janusgraph-core .jar , internally this is the method that gets called in StandardJanusGraph.class

If you see the below code highlighted  , even if their are errors during ES commit , nothing is re-thrown , those errors stored in MAP and after that those are printed as log statements . this leads data inconsistency , because in same code below where cassandra commit when their are any exceptions transaction is getting rollback , so that the changes are being rollback , but for ES this is not happening .

When we see the latest code for the StandardJanusGraph.class , their also no exception re-thrown , and it mentioned in the code this needs to be cleaned , does that means this is something not implemented completely ?

https://github.com/JanusGraph/janusgraph/blob/6bb2ba926b6cac2669f608f9461177d964ae0be0/janusgraph-core/src/main/java/org/janusgraph/graphdb/database/StandardJanusGraph.java#L761

is this issue with janusgraph core jars ? is there any fix happened already ?

public void commit(Collection<InternalRelation> addedRelations, Collection<InternalRelation> deletedRelations, StandardJanusGraphTx tx) {
    if (!addedRelations.isEmpty() || !deletedRelations.isEmpty()) {
        log.debug("Saving transaction. Added {}, removed {}", addedRelations.size(), deletedRelations.size());
        if (!tx.getConfiguration().hasCommitTime()) {
            tx.getConfiguration().setCommitTime(this.times.getTime());
        }

        Instant txTimestamp = tx.getConfiguration().getCommitTime();
        long transactionId = this.txCounter.incrementAndGet();
        if (!tx.getConfiguration().hasAssignIDsImmediately()) {
            this.idAssigner.assignIDs(addedRelations);
        }

        BackendTransaction mutator = tx.getTxHandle();
        boolean acquireLocks = tx.getConfiguration().hasAcquireLocks();
        boolean hasTxIsolation = this.backend.getStoreFeatures().hasTxIsolation();
        boolean logTransaction = this.config.hasLogTransactions() && !tx.getConfiguration().hasEnabledBatchLoading();
        KCVSLog txLog = logTransaction ? this.backend.getSystemTxLog() : null;
        TransactionLogHeader txLogHeader = new TransactionLogHeader(transactionId, txTimestamp, this.times);

        try {
            if (logTransaction) {
                Preconditions.checkNotNull(txLog, "Transaction log is null");
                txLog.add(txLogHeader.serializeModifications(this.serializer, LogTxStatus.PRECOMMIT, tx, addedRelations, deletedRelations), txLogHeader.getLogKey());
            }

            boolean hasSchemaElements = !Iterables.isEmpty(Iterables.filter(deletedRelations, SCHEMA_FILTER)) || !Iterables.isEmpty(Iterables.filter(addedRelations, SCHEMA_FILTER));
            Preconditions.checkArgument(!hasSchemaElements || !tx.getConfiguration().hasEnabledBatchLoading() && acquireLocks, "Attempting to create schema elements in inconsistent state");
            StandardJanusGraph.ModificationSummary commitSummary;
            if (hasSchemaElements && !hasTxIsolation) {
                BackendTransaction schemaMutator = this.openBackendTransaction(tx);

                try {
                    commitSummary = this.prepareCommit(addedRelations, deletedRelations, SCHEMA_FILTER, schemaMutator, tx, acquireLocks);

                    assert commitSummary.hasModifications && !commitSummary.has2iModifications;
                } catch (Throwable var42) {
                    schemaMutator.rollback();
                    throw var42;
                }

                try {
                    schemaMutator.commit();
                } catch (Throwable var40) {
                    log.error("Could not commit transaction [" + transactionId + "] due to storage exception in system-commit", var40);
                    throw var40;
                }
            }

            commitSummary = this.prepareCommit(addedRelations, deletedRelations, hasTxIsolation ? NO_FILTER : NO_SCHEMA_FILTER, mutator, tx, acquireLocks);
            if (commitSummary.hasModifications) {
                String logTxIdentifier = tx.getConfiguration().getLogIdentifier();
                boolean hasSecondaryPersistence = logTxIdentifier != null || commitSummary.has2iModifications;
                if (logTransaction) {
                    txLog.add(txLogHeader.serializePrimary(this.serializer, hasSecondaryPersistence ? LogTxStatus.PRIMARY_SUCCESS : LogTxStatus.COMPLETE_SUCCESS), txLogHeader.getLogKey(), mutator.getTxLogPersistor());
                }

                try {
                    mutator.commitStorage();
                } catch (Throwable var39) {
                    log.error("Could not commit transaction [" + transactionId + "] due to storage exception in commit", var39);
                    throw var39;
                }

                if (hasSecondaryPersistence) {
                    LogTxStatus status = LogTxStatus.SECONDARY_SUCCESS;
                    Map<String, Throwable> indexFailures = ImmutableMap.of();
                    boolean userlogSuccess = true;

                    try {
                        indexFailures = mutator.commitIndexes();
                        if (!((Map)indexFailures).isEmpty()) {
                            status = LogTxStatus.SECONDARY_FAILURE;
                            Iterator var20 = ((Map)indexFailures).entrySet().iterator();

                            while(var20.hasNext()) {
                                java.util.Map.Entry<String, Throwable> entry = (java.util.Map.Entry)var20.next();
                                log.error("Error while committing index mutations for transaction [" + transactionId + "] on index: " + (String)entry.getKey(), (Throwable)entry.getValue());
                            }
                        }

                        if (logTxIdentifier != null) {
                            try {
                                userlogSuccess = false;
                                Log userLog = this.backend.getUserLog(logTxIdentifier);
                                Future<Message> env = userLog.add(txLogHeader.serializeModifications(this.serializer, LogTxStatus.USER_LOG, tx, addedRelations, deletedRelations));
                                if (env.isDone()) {
                                    try {
                                        env.get();
                                    } catch (ExecutionException var37) {
                                        throw var37.getCause();
                                    }
                                }

                                userlogSuccess = true;
                            } catch (Throwable var38) {
                                status = LogTxStatus.SECONDARY_FAILURE;
                                log.error("Could not user-log committed transaction [" + transactionId + "] to " + logTxIdentifier, var38);
                            }
                        }
                    } finally {
                        if (logTransaction) {
                            try {
                                txLog.add(txLogHeader.serializeSecondary(this.serializer, status, (Map)indexFailures, userlogSuccess), txLogHeader.getLogKey());
                            } catch (Throwable var36) {
                                log.error("Could not tx-log secondary persistence status on transaction [" + transactionId + "]", var36);
                            }
                        }

                    }
                } else {
                    mutator.commitIndexes();
                }
            } else {
                mutator.commit();
            }

        } catch (Throwable var43) {
            log.error("Could not commit transaction [" + transactionId + "] due to exception", var43);

            try {
                mutator.rollback();
            } catch (Throwable var35) {
                log.error("Could not roll-back transaction [" + transactionId + "] after failure due to exception", var35);
            }

            if (var43 instanceof RuntimeException) {
                throw (RuntimeException)var43;
            } else {
                throw new JanusGraphException("Unexpected exception", var43);
            }
        }
    }
}


Re: The memory can only be add() during vertex program execute

Oleksandr Porunov <alexand...@...>
 

Hi Anjani,

This is JanusGraph developers channel to discuss JanusGraph development process. You can get help with your issue in the next channel: https://groups.google.com/forum/#!forum/janusgraph-users


On Wednesday, April 29, 2020 at 10:27:53 AM UTC-7, anj...@... wrote:
Hi All,

We have a custom vertex program which find connected nodes. We have some unique requirement to remove some duplicate data, for that we are collecting all data using collectAsMap() method.
But getting below mentioned error when load is more. For low load it works fine. 


Data Store: Cassandra
Index Store: Elasticsearch

Data :  ~300M Nodes & ~371M Edges. 
Spark Cluster: 100 Executors, 7 Cores



Job aborted due to stage failure: Task 199 in stage 14.0 failed 4 times, most recent failure: Lost task 199.3 in stage 14.0 (TID 9825, <ip>, executor 97): java.lang.IllegalArgumentException: The memory can only be add() during vertex program execute: gremlin.connectedNodeVertexProgram.voteToHalt
	at org.apache.tinkerpop.gremlin.process.computer.Memory$Exceptions.memoryAddOnlyDuringVertexProgramExecute(Memory.java:161)



I am not able to figure out why this exception is thrown, i don't see any OOM error. Please share thought/pointers.


Appreciate you help.


Thanks,

Anjani




The memory can only be add() during vertex program execute

anjani...@...
 

Hi All,

We have a custom vertex program which find connected nodes. We have some unique requirement to remove some duplicate data, for that we are collecting all data using collectAsMap() method.
But getting below mentioned error when load is more. For low load it works fine. 


Data Store: Cassandra
Index Store: Elasticsearch

Data :  ~300M Nodes & ~371M Edges. 
Spark Cluster: 100 Executors, 7 Cores



Job aborted due to stage failure: Task 199 in stage 14.0 failed 4 times, most recent failure: Lost task 199.3 in stage 14.0 (TID 9825, <ip>, executor 97): java.lang.IllegalArgumentException: The memory can only be add() during vertex program execute: gremlin.connectedNodeVertexProgram.voteToHalt
	at org.apache.tinkerpop.gremlin.process.computer.Memory$Exceptions.memoryAddOnlyDuringVertexProgramExecute(Memory.java:161)



I am not able to figure out why this exception is thrown, i don't see any OOM error. Please share thought/pointers.


Appreciate you help.


Thanks,

Anjani



201 - 220 of 1585