Reproduce OCL Benchmark

[schwichti]

Hi,

I have found the paper G. Daniel, G. Sunyé, and J. Cabot: "Mogwaï: a Framework to Handle Complex Queries on Large Models". I would like to reproduce the OCL benchmarks with Mogwai. The benchmarks can be found here: https://github.com/atlanmod/Mogwai/tree/master/benchmarks/fr.inria.atlanmod.mogwai.benchmarks
However, it seems that the benchmarks are inconsistent with the Mogwai API as some types like MogwaiOCLQueryBuilder, MogwaiQueryResult have been refactored.

Then I have found the OCL test cases provided here: https://github.com/atlanmod/Mogwai/tree/master/fr.inria.atlanmod.mogwai.transformation.ocl.tests
These test cases do not have compile errors. So I merged the OCL benchmarks into these test cases to get them running. I also copied the XMI models from here https://github.com/atlanmod/Mogwai/tree/master/benchmarks/fr.inria.atlanmod.mogwai.benchmarks/resources.

This is how my test case looks like:

public class RunMogwaiQueryTest {

  private static MogwaiResource resource;
  @BeforeClass
  public static void setUp() throws IOException {

    JavaPackage.eINSTANCE.eClass();
    EPackage.Registry.INSTANCE.put(JavaPackage.eNS_URI, JavaPackage.eINSTANCE);
    PersistenceBackendFactoryRegistry.register(MogwaiURI.MOGWAI_SCHEME,
        BlueprintsPersistenceBackendFactory.getInstance());
    ResourceSet rSet = new ResourceSetImpl();
    rSet.getResourceFactoryRegistry().getProtocolToFactoryMap()
    .put(MogwaiURI.MOGWAI_SCHEME, MogwaiResourceFactory.getInstance());

    URI xmiURI = URI.createURI("resources/models/org.eclipse.gmt.modisco.java.kyanos.xmi");
    URI mogwaiURI = MogwaiURI.createMogwaiURI(new File("resources/db/set1.graphdb"));

    if(!new File("resources/db/set1.graphdb").exists()) {

      try {
        ModelImporter.createNeoMogwaiResourceFromXMI(xmiURI, mogwaiURI);
      } catch (IOException e) {
        MogwaiLogger.error("Cannot create the Mogwai resource ({0})", mogwaiURI);
        throw e;
      }
    }

    resource = (MogwaiResource) rSet.createResource(MogwaiURI.createMogwaiURI(new File(
        "resources/db/set1.graphdb")));

    resource.load(Collections.emptyMap());
  }

  @Test
	public void textElementInJavadocQuery(){


		String expression = "self.compilationUnits.commentList->"
                + "select(each | each.oclIsTypeOf(Javadoc))->"
                + "collect(o | o.oclAsType(Javadoc).tags).fragments->"
                + "select(each | each.oclIsTypeOf(TextElement))->"
                + "asSequence()";		


		MogwaiQuery query = OCLQueryBuilder.newBuilder().fromString(expression)
				.context(JavaPackage.eINSTANCE.getModel()).build();
		NeoEMFQueryResult result = resource.query(query);

		System.out.println("mogwai: textElementInJavadocQuery size: "+result.getResults().size());
	}  

}

I can execute this as a regular JUnit Test (not a plugin test). In this case I get the following error:

javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: self for class: Script2
	at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:333)
	at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:41)
	at javax.script.CompiledScript.eval(Unknown Source)
	at fr.inria.atlanmod.mogwai.processor.GremlinScriptRunner.runGremlinScript(GremlinScriptRunner.java:120)
	at fr.inria.atlanmod.mogwai.processor.AbstractQueryProcessor.runGremlinScript(AbstractQueryProcessor.java:370)
	at fr.inria.atlanmod.mogwai.processor.AbstractQueryProcessor.process(AbstractQueryProcessor.java:199)
	at fr.inria.atlanmod.mogwai.processor.AbstractATLProcessor.process(AbstractATLProcessor.java:70)
	at fr.inria.atlanmod.mogwai.processor.OCLQueryProcessor.process(OCLQueryProcessor.java:71)
	at fr.inria.atlanmod.mogwai.neoemf.processor.NeoEMFOCLQueryProcessor.process(NeoEMFOCLQueryProcessor.java:68)
	at fr.inria.atlanmod.mogwai.processor.OCLQueryProcessor.process(OCLQueryProcessor.java:1)
	at fr.inria.atlanmod.mogwai.query.MogwaiQuery.process(MogwaiQuery.java:99)
	at fr.inria.atlanmod.mogwai.query.MogwaiQuery.process(MogwaiQuery.java:74)
	at fr.inria.atlanmod.mogwai.neoemf.util.NeoEMFQueryHandler.query(NeoEMFQueryHandler.java:158)
	at fr.inria.atlanmod.mogwai.neoemf.resource.DefaultMogwaiResource.query(DefaultMogwaiResource.java:92)
	at fr.inria.atlanmod.mogwai.neoemf.resource.MogwaiResource.query(MogwaiResource.java:119)
	at fr.inria.atlanmod.mogwai.transformation.ocl.tests.execution.RunMogwaiQueryTest.textElementInJavadocQuery(RunMogwaiQueryTest.java:105)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
Caused by: groovy.lang.MissingPropertyException: No such property: self for class: Script2
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:50)
	at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:49)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:231)
	at Script2.run(Script2.groovy:5)
	at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:330)
	... 39 more

It seems to me that OCL expressions containing self reference are not supported right now (or that the context is not set properly). So I modified the expression like this:

String expression = "Model.allInstances()->any(true).compilationUnits.commentList->"
            + "select(each | each.oclIsTypeOf(Javadoc))->"
            + "collect(o | o.oclAsType(Javadoc).tags).fragments->"
            + "select(each | each.oclIsTypeOf(TextElement))->"
            + "asSequence()";

This seem working, although I see the following error:

javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method inE() on null object
	at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:333)
	at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:41)
	at javax.script.CompiledScript.eval(Unknown Source)
	at fr.inria.atlanmod.mogwai.processor.GremlinScriptRunner.runGremlinScript(GremlinScriptRunner.java:120)
	at fr.inria.atlanmod.mogwai.processor.AbstractQueryProcessor.runGremlinScript(AbstractQueryProcessor.java:370)
	at fr.inria.atlanmod.mogwai.processor.AbstractQueryProcessor.process(AbstractQueryProcessor.java:199)
	at fr.inria.atlanmod.mogwai.processor.AbstractATLProcessor.process(AbstractATLProcessor.java:70)
	at fr.inria.atlanmod.mogwai.processor.OCLQueryProcessor.process(OCLQueryProcessor.java:71)
	at fr.inria.atlanmod.mogwai.neoemf.processor.NeoEMFOCLQueryProcessor.process(NeoEMFOCLQueryProcessor.java:68)
	at fr.inria.atlanmod.mogwai.processor.OCLQueryProcessor.process(OCLQueryProcessor.java:1)
	at fr.inria.atlanmod.mogwai.query.MogwaiQuery.process(MogwaiQuery.java:99)
	at fr.inria.atlanmod.mogwai.query.MogwaiQuery.process(MogwaiQuery.java:74)
	at fr.inria.atlanmod.mogwai.neoemf.util.NeoEMFQueryHandler.query(NeoEMFQueryHandler.java:158)
	at fr.inria.atlanmod.mogwai.neoemf.resource.DefaultMogwaiResource.query(DefaultMogwaiResource.java:92)
	at fr.inria.atlanmod.mogwai.neoemf.resource.MogwaiResource.query(MogwaiResource.java:119)
	at fr.inria.atlanmod.mogwai.transformation.ocl.tests.execution.RunMogwaiQueryTest.grabats09(RunMogwaiQueryTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
Caused by: java.lang.NullPointerException: Cannot invoke method inE() on null object
	at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:77)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:45)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
	at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:32)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
	at Script2.run(Script2.groovy:5)
	at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:330)
	... 39 more

The following tables show the result sizes reported in the paper for the MoDisco and JDT test model and what I have reproduced.

MoDisco:

Testcase	Paper	Reproduced Mogwai	Reproduced OCL Interpreter	Reproduced IncQuery/Viatra
textElementInJavadoc	12359	12359 ✔️	12359 ✔️	12359 ✔️
throwsExceptions	0	0 ✔️	0 ✔️	0 ✔️
invisibleMethods	134	0 ❌	134 ✔️	134 ✔️
grabats09	0	0 ✔️	0 ✔️	0 ✔️

JDT:

Testcase	Paper	Reproduced Mogwai	Reproduced OCL Interpreter	Reproduced IncQuery/Viatra
textElementInJavadoc	54201	54201 ✔️	54201 ✔️	54201 ✔️
throwsExceptions	1155	1155 ✔️	1155 ✔️	1155 ✔️
invisibleMethods	3927	0 ❌	3927 ✔️	3927 ✔️
grabats09	92	0 ❌	42 ❌	42 ❌

I am grateful for any support to reproduce the results reported in the paper.

[gdaniel]

Hi,

Thanks for your interest in the tool! First of all, I have to confess that I haven't seriously worked on Mogwaï in the last months (at least not for implementing new features), so my memory may be a bit blurry for some specific points, I'll put here what should be working, and what shouldn't, so you won't waste your time, and I'll come back to you next week with a more detailed answer.

1. Indeed, the benchmarks are not consistent anymore with the new API. I created the benchmarks for the paper you read, but I entirely refactored the engine for this paper in order to support multiple input languages (currently OCL and and experimental ATL support), and I didn't take the time to update the old benchmarks.

2. The self variable should be working, but you need to provide an additional parameter representing the self element to the MogwaiResource#query. In your case this parameter should be the instance of top-level Model element, so you should use something like resource.query(query, resource.getContents().get(0)).

3. I am pretty sure I forgot to implement the mapping rule for any, I'll double-check, but I won't be surprise that your exception comes from a null instance generated by the engine for the input any operation, that cannot be chained to the rest of the query.

4. I'll have a closer look at the benchmarks, and I'll try to adapt them to the new architecture, and see if I can reproduce your errors. I am a bit surprised by the reproduced OCL interpreter for grabats09 though, but I remember I had some issues in the past with this query.

I hope this answer helps you to understand some of the issues. As I said, I'll try to have a look and fix what can be "easily" fixed next week, and come back to you with more information.

Regards,

Gwendal

[schwichti]

Many thanks for your support. I also tried to reproduce the results with Viatra/IncQuery. I have updated the table above accordingly. The result size for grabats09 is also 42 and not 92 as reported in the paper.

Additionally, I had to fix the pattern thrownExceptions as follows:

pattern thrownExceptions(Ex) {
	//Package(Pack); removed
	//Package.ownedElements(Pack,Owned); removed
	ClassDeclaration(Owned);
	ClassDeclaration.bodyDeclarations(Owned,Body);
	MethodDeclaration(Body);
	MethodDeclaration.thrownExceptions(Body,Ex);
}

[schwichti]

It seems that the IncQuery/Viatra pattern "emptyTextElementInJavadoc" is missing in this repository. So I have recreated it:

pattern emptyTextElementInJavadoc(Ff) {
	Model(Mm);
	Model.compilationUnits.commentList(Mm,Cl);
	Javadoc(Cl);
	Javadoc.tags.fragments(Cl,Ff);
	TextElement(Ff);
	TextElement.text(Ff, x);
	check(x.length()==0);
}

This pattern gives me the expected result sizes.

[gdaniel]

Hi, you can pull the last version of the repository that should fix your issues. For the record, I am able to run the JDT queries and retrieve the expected results (I didn't try on MoDisco though, if you have any issue with this model let me know).

I didn't investigate that much in the Grabats query, I don't know why it is returning 42 results while the paper states 92 (even if 42 should be the answer for all the queries ;)). I think it was related to a missing flatten() operation somewhere, but I cannot retrieve it. Anyway, the Mogwaï results are now consistent with the OCL ones, so it is a matter of retrieving the OCL query I used for the paper, I'll update the repository with it later if I find it.

Thanks for the update on IncQuery, I'll add it to the benchmarks.

Please note that the tools has evolved a lot since the publication of this article, and the execution time and memory consumption results can differ from what I reported. If you observe a major issue on performances and memory consumption let me know!

Cheers,

Gwendal

[schwichti]

Thank you very much. The test cases are passing.
However, when I replace

public void run() {
	MogwaiQuery query = OCLQueryBuilder.newBuilder().fromURI(URI.createURI("ocl/RCIS/ThrownExceptions.ocl"))
				.build();
	NeoLogger.info("Input Query: {0}" + query.getInput());
        startTimer();
        MogwaiResource mogwaiResource = (MogwaiResource)resource;
        NeoEMFQueryResult result = mogwaiResource.query(query);
        endTimer();
        NeoLogger.info("Result size: {0}", result.resultSize());
}

by

public void run(){
	
	String expression = "ClassDeclaration.allInstances()->"
               + "collect(bodyDeclarations)->"
               + "select(each | each.oclIsTypeOf(MethodDeclaration))->"
               + "collect(each | each.oclAsType(MethodDeclaration).thrownExceptions)"
               + "->asSequence()";
		
	MogwaiQuery query = OCLQueryBuilder.newBuilder().fromString(expression)
			.context(JavaPackage.eINSTANCE.getModel()).build();
	MogwaiResource mogwaiResource = (MogwaiResource)resource;
	NeoEMFQueryResult result = mogwaiResource.query(query);
}

I see the following Exception:

java.lang.ExceptionInInitializerError
	at fr.inria.atlanmod.mogwai.benchmarks.mogwai.tests.ThrownExceptionsQuery.run(ThrownExceptionsQuery.java:57)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:538)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:760)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:460)
	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:206)
Caused by: java.lang.RuntimeException: Missing serialized package: java.ecore
	at org.eclipse.gmt.modisco.java.emf.impl.JavaPackageImpl.loadPackage(JavaPackageImpl.java:4572)
	at org.eclipse.gmt.modisco.java.emf.impl.JavaPackageImpl.init(JavaPackageImpl.java:1029)
	at org.eclipse.gmt.modisco.java.emf.JavaPackage.<clinit>(JavaPackage.java:75)
	... 35 more

[schwichti]

When I copy java.ecore manually to /bin/org/eclipse/gmt/modisco/java/emf/impl it works!

[gdaniel]

Great to hear you managed to make it work! I'll have a look on this, I have seen some weird dependencies to the bin folder that should not be there.
I don't close your issue until I solve this dependency error, so I'll not forget it when I come back from holidays.
If you have another issue with the benchmark let me know!

Cheers,

Gwendal