Include tests generated by Botsing/Evosuite in our build

Hi devs,

As part of STAMP we have a new tool that generates tests (K09 KPI):

  • Botsing generates a model of our code
  • Then Evosuite is executed on that model and generates tests

I’ve tested it on XMLUtils and found that the generated test do increase the mutation score + coverage, even though I don’t find the tests wildly interesting. They could still be interesting to run as part of our build (similar to dspot-generated tests). FYI here’s what was generated for XMLUtils: https://gist.github.com/vmassol/191fbd9f2269f830362daae66af75848

In this example, the tests raised the coverage from 64.5% to 71.4% and the mutation score from 65% to 76%.

So I’m proposing to add these type of tests in a src/test/evosuite (or src/test/botsing) directory in our sources, similar to what we do for DSpot already.

WDYT?

Thanks

+1 it’s not hurting to have more tests

+1

Explanations to run Botsing/Evosuite on XWiki modules

First time and until new versions of Botsing/Evosuite are released

  • git clone https://github.com/STAMP-project/evosuite.git
  • mvn clean install -DskipTests
  • git clone https://github.com/STAMP-project/botsing.git
  • Modify pom.xml
diff --git a/pom.xml b/pom.xml
index a693ebf..06ede8d 100644
--- a/pom.xml
+++ b/pom.xml
@@ -72,7 +72,7 @@
         <commons-cli.version>1.4</commons-cli.version>
         <!-- <coveralls-maven-plugin.version>4.3.0</coveralls-maven-plugin.version> -->
         <evosuite.version>1.0.6</evosuite.version>
-        <evosuite-client.version>1.0.7</evosuite-client.version>
+        <evosuite-client.version>1.0.7-SNAPSHOT</evosuite-client.version>
         <guava.version>27.0.1-jre</guava.version>
 
         <hamcrest.version>2.0.0.0</hamcrest.version>
@@ -421,4 +421,4 @@
         <module>botsing-parsers</module>
     </modules>
 
-</project>
\ No newline at end of file
+</project>
  • Install jar in local maven repo: cd botsing-reproduction/ and mvn install:install-file -Dfile=evosuite-client-botsing-1.0.7.jar -DgroupId=org.evosuite -DartifactId=evosuite-client-botsing -Dversion=1.0.7-SNAPSHOT -Dpackaging=jar
  • Build botsing: mvn clean install -DskipTests

To execute on a xwiki module

  • Cd to the module director, for example for Commons XML: cd xwiki-commons-xml.
  • Set CP variable. Run mvn dependency:build-classpath and copy results in $xwikicp shell variable and add target/classes and target/test-classestoo:
export xwikicp="target/classes:target/test-classes:<copied path>"
  • Execute botsing seeding to generate a model of the code: java -d64 -Xmx10000m -jar ~/.m2/repository/eu/stamp-project/botsing-model-generation/1.0.8-SNAPSHOT/botsing-model-generation-1.0.8-SNAPSHOT-jar-with-dependencies.jar -project_cp "$xwikicp" -project_prefix "org.xwiki.xml" -out_dir target/botsing
    • Note that Botsing Seeding has 2 phases to generate the model: a static analysis phase and a dynamic analysis one (by executing the tests). However, currently Botsing/Evosuite don’t support JUnit5 tests so the dynamic phase won’t execute the JUnit5 tests. The model is still generated but is a bit less good that what it could be.
  • Execute Evosuite to generate the tests (notice the -class parameter to specify for which class to generate tests for):
  java -d64 -Xmx4000m -jar ~/.m2/repository/org/evosuite/evosuite-master/1.0.7-SNAPSHOT/evosuite-master-1.0.7-SNAPSHOT.jar \
    -target "target/classes" \
  	-projectCP "$xwikicp" \
  	-generateMOSuite \
  	-Dalgorithm=DynaMOSA \
  	-Dsearch_budget=60 \
  	-Dseed_clone="0.5" \
  	-Donline_model_seeding=TRUE \
  	-Dmodel_path="target/botsing/models" \
  	-Dtest_dir="target/botsing/evosuite-tests" \
  	-Dreport_dir="target/botsing/evosuite-report" \
  	-Dno_runtime_dependency=true
  • Note that you can pass -prefix "org.xwiki.xml" to generate tests for all classes in the passed package.
  • Generated tests can be found in target/botsing/evosuite-tests.

For history, see https://github.com/STAMP-project/botsing/issues/89

Done with https://jira.xwiki.org/browse/XCOMMONS-1759

+1

Note that Botsing/Evosuite considers that the existing code is correct and generates tests that pass. This means that there’s a bug in our code, the tests will pass, considering that the bug is a normal behavior, and it’ll even increase the coverage and mutation score :slight_smile: Ideally the tests should be reviewed but since this is painful I think we should just add them and when we fix the bug, the generated tests will fail and we’ll just need to regenerate them. This means the global coverage and mutation score could actually decrease as a result.

For example just noticed this generated test:

@Test(timeout = 4000)
  public void test5()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      StringReader stringReader0 = new StringReader("'");
      // Undeclared exception!
      try { 
        defaultHTMLCleaner0.clean((Reader) stringReader0);
        fail("Expecting exception: NullPointerException");
      
      } catch(NullPointerException e) {
         //
         // no message in exception (getMessage() returned null)
         //
      }
  }

Doesn’t seem that good…

Actually most if not all the tests in this generated test class don’t look that good:

/*
 * This file was automatically generated by EvoSuite
 * Mon Oct 07 13:18:11 GMT 2019
 */

package org.xwiki.xml.internal.html;

import org.junit.Test;
import static org.junit.Assert.*;
import static org.evosuite.shaded.org.mockito.Mockito.*;
import java.io.Reader;
import java.io.StringReader;
import org.evosuite.runtime.ViolatedAssumptionAnswer;
import org.evosuite.runtime.javaee.injection.Injector;
import org.w3c.dom.Document;
import org.xwiki.context.Execution;
import org.xwiki.context.ExecutionContext;
import org.xwiki.xml.html.HTMLCleanerConfiguration;
import org.xwiki.xml.internal.html.DefaultHTMLCleaner;
import org.xwiki.xml.internal.html.DefaultHTMLCleanerConfiguration;
import org.xwiki.xml.internal.html.filter.FontFilter;
import org.xwiki.xml.internal.html.filter.ListFilter;
import org.xwiki.xml.internal.html.filter.UniqueIdFilter;

public class DefaultHTMLCleaner_ESTest {

  @Test(timeout = 4000)
  public void test0()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      // Undeclared exception!
      try { 
        defaultHTMLCleaner0.clean((Reader) null);
        fail("Expecting exception: RuntimeException");
      
      } catch(RuntimeException e) {
         //
         // Unhandled error when cleaning HTML
         //
      }
  }

  @Test(timeout = 4000)
  public void test1()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      UniqueIdFilter uniqueIdFilter0 = new UniqueIdFilter();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "attributeFilter", (Object) uniqueIdFilter0);
      FontFilter fontFilter0 = new FontFilter();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "bodyFilter", (Object) fontFilter0);
      Execution execution0 = mock(Execution.class, new ViolatedAssumptionAnswer());
      doReturn((ExecutionContext) null).when(execution0).getContext();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "execution", (Object) execution0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "fontFilter", (Object) fontFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "linkFilter", (Object) fontFilter0);
      ListFilter listFilter0 = new ListFilter();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "listFilter", (Object) listFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "listItemFilter", (Object) listFilter0);
      Injector.validateBean(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class);
      StringReader stringReader0 = new StringReader("Eb_H#\"[M");
      Document document0 = defaultHTMLCleaner0.clean((Reader) stringReader0);
      assertNotNull(document0);
  }

  @Test(timeout = 4000)
  public void test2()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      UniqueIdFilter uniqueIdFilter0 = new UniqueIdFilter();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "attributeFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "bodyFilter", (Object) uniqueIdFilter0);
      ExecutionContext executionContext0 = new ExecutionContext();
      Execution execution0 = mock(Execution.class, new ViolatedAssumptionAnswer());
      doReturn(executionContext0).when(execution0).getContext();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "execution", (Object) execution0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "fontFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "linkFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "listFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "listItemFilter", (Object) uniqueIdFilter0);
      Injector.validateBean(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class);
      StringReader stringReader0 = new StringReader("\b K.}|3}@L<U(E|Gpi");
      Document document0 = defaultHTMLCleaner0.clean((Reader) stringReader0);
      assertNotNull(document0);
  }

  @Test(timeout = 4000)
  public void test3()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      HTMLCleanerConfiguration hTMLCleanerConfiguration0 = defaultHTMLCleaner0.getDefaultConfiguration();
      // Undeclared exception!
      try { 
        defaultHTMLCleaner0.clean((Reader) null, hTMLCleanerConfiguration0);
        fail("Expecting exception: RuntimeException");
      
      } catch(RuntimeException e) {
         //
         // Unhandled error when cleaning HTML
         //
      }
  }

  @Test(timeout = 4000)
  public void test4()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      UniqueIdFilter uniqueIdFilter0 = new UniqueIdFilter();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "attributeFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "bodyFilter", (Object) uniqueIdFilter0);
      ExecutionContext executionContext0 = mock(ExecutionContext.class, new ViolatedAssumptionAnswer());
      doReturn(uniqueIdFilter0).when(executionContext0).getProperty(anyString());
      Execution execution0 = mock(Execution.class, new ViolatedAssumptionAnswer());
      doReturn(executionContext0).when(execution0).getContext();
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "execution", (Object) execution0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "fontFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "linkFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "listFilter", (Object) uniqueIdFilter0);
      Injector.inject(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class, "listItemFilter", (Object) uniqueIdFilter0);
      Injector.validateBean(defaultHTMLCleaner0, (Class<?>) DefaultHTMLCleaner.class);
      StringReader stringReader0 = new StringReader("\b K.}|3}@L<U(E|Gpi");
      // Undeclared exception!
      try { 
        defaultHTMLCleaner0.clean((Reader) stringReader0);
        fail("Expecting exception: ClassCastException");
      
      } catch(ClassCastException e) {
         //
         // org.xwiki.xml.internal.html.filter.UniqueIdFilter cannot be cast to javax.xml.parsers.DocumentBuilder
         //
      }
  }

  @Test(timeout = 4000)
  public void test5()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      StringReader stringReader0 = new StringReader("'");
      // Undeclared exception!
      try { 
        defaultHTMLCleaner0.clean((Reader) stringReader0);
        fail("Expecting exception: NullPointerException");
      
      } catch(NullPointerException e) {
         //
         // no message in exception (getMessage() returned null)
         //
      }
  }

  @Test(timeout = 4000)
  public void test6()  throws Throwable  {
      DefaultHTMLCleaner defaultHTMLCleaner0 = new DefaultHTMLCleaner();
      StringReader stringReader0 = new StringReader("namespacesAware");
      DefaultHTMLCleanerConfiguration defaultHTMLCleanerConfiguration0 = new DefaultHTMLCleanerConfiguration();
      // Undeclared exception!
      try { 
        defaultHTMLCleaner0.clean((Reader) stringReader0, (HTMLCleanerConfiguration) defaultHTMLCleanerConfiguration0);
        fail("Expecting exception: NullPointerException");
      
      } catch(NullPointerException e) {
         //
         // no message in exception (getMessage() returned null)
         //
      }
  }
}

About the NPEs I’ve opened https://github.com/STAMP-project/botsing/issues/97

Also created https://github.com/STAMP-project/botsing/issues/96

Another issue: some generated tests output content to stdout which should be caught by XWiki’s console checker (it’s not caught right now; I think it’s because we only check for junit5 but that’s strange since I was sure we were also checking for junit4… need to refresh my memory :)).

For example:

[INFO] Running org.xwiki.xml.XMLUtils_ESTest
[Fatal Error] :1:1: Content is not allowed in prolog.
ERROR:  'Content is not allowed in prolog.'
ERROR:  ''
[Fatal Error] :1:1: Content is not allowed in prolog.
15:35:01.467 [Time-limited test] WARN  org.xwiki.xml.XMLUtils - Cannot parse XML document: [Content is not allowed in prolog.]
15:35:01.519 [Time-limited test] WARN  org.xwiki.xml.XMLUtils - Failed to serialize node to XML String: [null]
ERROR:  'Could not compile stylesheet'
FATAL ERROR:  'The input document is not a stylesheet (the XSL namespace is not declared in the root element).'
           :The input document is not a stylesheet (the XSL namespace is not declared in the root element).
15:35:01.534 [Time-limited test] WARN  org.xwiki.xml.XMLUtils - Failed to apply XSLT transformation: [The input document is not a stylesheet (the XSL namespace is not declared in the root element).]
[INFO] Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.091 s - in org.xwiki.xml.XMLUtils_ESTest

Another issue: generated tests can create files outside the target directory. For example:

37

We can imagine situations when this is very dangerous: removing files from the local computer, etc…

The reason is actually because of XCOMMONS-1742: https://github.com/xwiki/xwiki-commons/commit/d2d0904b0c82e674f7ac1f022433b008ed186077#diff-90a6ff0201501b4b3520d18db75b7268L1

We had a TestListener before and this was working for both junit4 and junit5 (up to Junit 5.5). Then we converted the Listener to a JUnit5 Extension and at the same time we lost the log capture for JUnit4! Without realizing it…

There are too many problem with this strategy and the Botsing tool is not yet ready for this approach. Thus I’m withdrawing the proposal the reverting the commit.

Thx