changeset 52342:9341b077bd55

8210958: Rename "make run-test" to "make test" Reviewed-by: erikj
author ihse
date Wed, 31 Oct 2018 09:30:24 +0100
parents 2b58b8e1d28f
children b66b51c4e405
files doc/testing.html doc/testing.md make/Help.gmk make/Main.gmk make/MainSupport.gmk make/RunTests.gmk make/common/FindTests.gmk make/conf/jib-profiles.js test/Makefile test/make/TestMake.gmk test/make/TestMakeBase.gmk
diffstat 11 files changed, 175 insertions(+), 126 deletions(-) [+]
line wrap: on
line diff
--- a/doc/testing.html	Wed Oct 31 08:09:45 2018 +0100
+++ b/doc/testing.html	Wed Oct 31 09:30:24 2018 +0100
@@ -18,12 +18,13 @@
 </header>
 <nav id="TOC">
 <ul>
-<li><a href="#using-the-run-test-framework">Using the run-test framework</a><ul>
+<li><a href="#using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</a><ul>
 <li><a href="#configuration">Configuration</a></li>
 </ul></li>
 <li><a href="#test-selection">Test selection</a><ul>
 <li><a href="#jtreg">JTReg</a></li>
 <li><a href="#gtest">Gtest</a></li>
+<li><a href="#special-tests">Special tests</a></li>
 </ul></li>
 <li><a href="#test-results-and-summary">Test results and summary</a></li>
 <li><a href="#test-suite-control">Test suite control</a><ul>
@@ -32,22 +33,23 @@
 </ul></li>
 </ul>
 </nav>
-<h2 id="using-the-run-test-framework">Using the run-test framework</h2>
+<h2 id="using-make-test-the-run-test-framework">Using &quot;make test&quot; (the run-test framework)</h2>
 <p>This new way of running tests is developer-centric. It assumes that you have built a JDK locally and want to test it. Running common test targets is simple, and more complex ad-hoc combination of tests is possible. The user interface is forgiving, and clearly report errors it cannot resolve.</p>
-<p>The main target &quot;run-test&quot; uses the jdk-image as the tested product. There is also an alternate target &quot;exploded-run-test&quot; that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
+<p>The main target <code>test</code> uses the jdk-image as the tested product. There is also an alternate target <code>exploded-test</code> that uses the exploded image instead. Not all tests will run successfully on the exploded image, but using this target can greatly improve rebuild times for certain workflows.</p>
+<p>Previously, <code>make test</code> was used invoke an old system for running test, and <code>make run-test</code> was used for the new test framework. For backward compatibility with scripts and muscle memory, <code>run-test</code> (and variants like <code>exploded-run-test</code> or <code>run-test-tier1</code>) are kept as aliases. The old system can still be accessed for some time using <code>cd test &amp;&amp; make</code>.</p>
 <p>Some example command-lines:</p>
-<pre><code>$ make run-test-tier1
-$ make run-test-jdk_lang JTREG=&quot;JOBS=8&quot;
-$ make run-test TEST=jdk_lang
-$ make run-test-only TEST=&quot;gtest:LogTagSet gtest:LogTagSetDescriptions&quot; GTEST=&quot;REPEAT=-1&quot;
-$ make run-test TEST=&quot;hotspot:hotspot_gc&quot; JTREG=&quot;JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug&quot;
-$ make run-test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot;
-$ make exploded-run-test TEST=tier2</code></pre>
+<pre><code>$ make test-tier1
+$ make test-jdk_lang JTREG=&quot;JOBS=8&quot;
+$ make test TEST=jdk_lang
+$ make test-only TEST=&quot;gtest:LogTagSet gtest:LogTagSetDescriptions&quot; GTEST=&quot;REPEAT=-1&quot;
+$ make test TEST=&quot;hotspot:hotspot_gc&quot; JTREG=&quot;JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug&quot;
+$ make test TEST=&quot;jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java&quot;
+$ make exploded-test TEST=tier2</code></pre>
 <h3 id="configuration">Configuration</h3>
 <p>To be able to run JTReg tests, <code>configure</code> needs to know where to find the JTReg test framework. If it is not picked up automatically by configure, use the <code>--with-jtreg=&lt;path to jtreg home&gt;</code> option to point to the JTReg framework. Note that this option should point to the JTReg home, i.e. the top directory, containing <code>lib/jtreg.jar</code> etc. (An alternative is to set the <code>JT_HOME</code> environment variable to point to the JTReg home before running <code>configure</code>.)</p>
 <h2 id="test-selection">Test selection</h2>
-<p>All functionality is available using the run-test make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, run-test-only can be used instead, which do not depend on the source and test image build.</p>
-<p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make run-test-tier1</code> is equivalent to <code>make run-test TEST=&quot;tier1&quot;</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>run-test TEST=&quot;x&quot;</code> solution needs to be used.</p>
+<p>All functionality is available using the <code>test</code> make target. In this use case, the test or tests to be executed is controlled using the <code>TEST</code> variable. To speed up subsequent test runs with no source code changes, <code>test-only</code> can be used instead, which do not depend on the source and test image build.</p>
+<p>For some common top-level tests, direct make targets have been generated. This includes all JTReg test groups, the hotspot gtest, and custom tests (if present). This means that <code>make test-tier1</code> is equivalent to <code>make test TEST=&quot;tier1&quot;</code>, but the latter is more tab-completion friendly. For more complex test runs, the <code>test TEST=&quot;x&quot;</code> solution needs to be used.</p>
 <p>The test specifications given in <code>TEST</code> is parsed into fully qualified test descriptors, which clearly and unambigously show which tests will be run. As an example, <code>:tier1</code> will expand to <code>jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1</code>. You can always submit a list of fully qualified test descriptors in the <code>TEST</code> variable if you want to shortcut the parser.</p>
 <h3 id="jtreg">JTReg</h3>
 <p>JTReg tests can be selected either by picking a JTReg test group, or a selection of files or directories containing JTReg tests.</p>
@@ -59,6 +61,14 @@
 <p>Since the Hotspot Gtest suite is so quick, the default is to run all tests. This is specified by just <code>gtest</code>, or as a fully qualified test descriptor <code>gtest:all</code>.</p>
 <p>If you want, you can single out an individual test or a group of tests, for instance <code>gtest:LogDecorations</code> or <code>gtest:LogDecorations.level_test_vm</code>. This can be particularly useful if you want to run a shaky test repeatedly.</p>
 <p>For Gtest, there is a separate test suite for each JVM variant. The JVM variant is defined by adding <code>/&lt;variant&gt;</code> to the test descriptor, e.g. <code>gtest:Log/client</code>. If you specify no variant, gtest will run once for each JVM variant present (e.g. server, client). So if you only have the server JVM present, then <code>gtest:all</code> will be equivalent to <code>gtest:all/server</code>.</p>
+<h3 id="special-tests">Special tests</h3>
+<p>A handful of odd tests that are not covered by any other testing framework are accessible using the <code>special:</code> test descriptor. Currently, this includes <code>hotspot-internal</code>, <code>failure-handler</code> and <code>make</code>.</p>
+<ul>
+<li><p>Hotspot legacy internal testing (run using <code>-XX:+ExecuteInternalVMTests</code>) is run using <code>special:hotspot-internal</code> or just <code>hotspot-internal</code> as test descriptor, and will only work on a debug JVM.</p></li>
+<li><p>Failure handler testing is run using <code>special:failure-handler</code> or just <code>failure-handler</code> as test descriptor.</p></li>
+<li><p>Tests for the build system, including both makefiles and related functionality, is run using <code>special:make</code> or just <code>make</code> as test descriptor. This is equivalent to <code>special:make:all</code>.</p>
+<p>A specific make test can be run by supplying it as argument, e.g. <code>special:make:idea</code>. As a special syntax, this can also be expressed as <code>make-idea</code>, which allows for command lines as <code>make test-make-idea</code>.</p></li>
+</ul>
 <h2 id="test-results-and-summary">Test results and summary</h2>
 <p>At the end of the test run, a summary of all tests run will be presented. This will have a consistent look, regardless of what test suites were used. This is a sample summary:</p>
 <pre><code>==============================
@@ -72,7 +82,7 @@
 TEST FAILURE</code></pre>
 <p>Tests where the number of TOTAL tests does not equal the number of PASSed tests will be considered a test failure. These are marked with the <code>&gt;&gt; ... &lt;&lt;</code> marker for easy identification.</p>
 <p>The classification of non-passed tests differs a bit between test suites. In the summary, ERROR is used as a catch-all for tests that neither passed nor are classified as failed by the framework. This might indicate test framework error, timeout or other problems.</p>
-<p>In case of test failures, <code>make run-test</code> will exit with a non-zero exit value.</p>
+<p>In case of test failures, <code>make test</code> will exit with a non-zero exit value.</p>
 <p>All tests have their result stored in <code>build/$BUILD/test-results/$TEST_ID</code>, where TEST_ID is a path-safe conversion from the fully qualified test descriptor, e.g. for <code>jtreg:jdk/test:tier1</code> the TEST_ID is <code>jtreg_jdk_test_tier1</code>. This path is also printed in the log at the end of the test run.</p>
 <p>Additional work data is stored in <code>build/$BUILD/test-support/$TEST_ID</code>. For some frameworks, this directory might contain information that is useful in determining the cause of a failed test.</p>
 <h2 id="test-suite-control">Test suite control</h2>
--- a/doc/testing.md	Wed Oct 31 08:09:45 2018 +0100
+++ b/doc/testing.md	Wed Oct 31 09:30:24 2018 +0100
@@ -1,26 +1,32 @@
 % Testing the JDK
 
-## Using the run-test framework
+## Using "make test" (the run-test framework)
 
 This new way of running tests is developer-centric. It assumes that you have
 built a JDK locally and want to test it. Running common test targets is simple,
 and more complex ad-hoc combination of tests is possible. The user interface is
 forgiving, and clearly report errors it cannot resolve.
 
-The main target "run-test" uses the jdk-image as the tested product. There is
-also an alternate target "exploded-run-test" that uses the exploded image
+The main target `test` uses the jdk-image as the tested product. There is
+also an alternate target `exploded-test` that uses the exploded image
 instead. Not all tests will run successfully on the exploded image, but using
 this target can greatly improve rebuild times for certain workflows.
 
+Previously, `make test` was used invoke an old system for running test, and
+`make run-test` was used for the new test framework. For backward compatibility
+with scripts and muscle memory, `run-test` (and variants like
+`exploded-run-test` or `run-test-tier1`) are kept as aliases. The old system
+can still be accessed for some time using `cd test && make`.
+
 Some example command-lines:
 
-    $ make run-test-tier1
-    $ make run-test-jdk_lang JTREG="JOBS=8"
-    $ make run-test TEST=jdk_lang
-    $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
-    $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
-    $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
-    $ make exploded-run-test TEST=tier2
+    $ make test-tier1
+    $ make test-jdk_lang JTREG="JOBS=8"
+    $ make test TEST=jdk_lang
+    $ make test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
+    $ make test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
+    $ make test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
+    $ make exploded-test TEST=tier2
 
 ### Configuration
 
@@ -33,16 +39,16 @@
 
 ## Test selection
 
-All functionality is available using the run-test make target. In this use
-case, the test or tests to be executed is controlled using the `TEST` variable.
-To speed up subsequent test runs with no source code changes, run-test-only can
-be used instead, which do not depend on the source and test image build.
+All functionality is available using the `test` make target. In this use case,
+the test or tests to be executed is controlled using the `TEST` variable. To
+speed up subsequent test runs with no source code changes, `test-only` can be
+used instead, which do not depend on the source and test image build.
 
 For some common top-level tests, direct make targets have been generated. This
 includes all JTReg test groups, the hotspot gtest, and custom tests (if
-present). This means that `make run-test-tier1` is equivalent to `make run-test
+present). This means that `make test-tier1` is equivalent to `make test
 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
-test runs, the `run-test TEST="x"` solution needs to be used.
+test runs, the `test TEST="x"` solution needs to be used.
 
 The test specifications given in `TEST` is parsed into fully qualified test
 descriptors, which clearly and unambigously show which tests will be run. As an
@@ -98,6 +104,27 @@
 variant present (e.g. server, client). So if you only have the server JVM
 present, then `gtest:all` will be equivalent to `gtest:all/server`.
 
+### Special tests
+
+A handful of odd tests that are not covered by any other testing framework are
+accessible using the `special:` test descriptor. Currently, this includes
+`hotspot-internal`, `failure-handler` and `make`.
+
+  * Hotspot legacy internal testing (run using `-XX:+ExecuteInternalVMTests`)
+    is run using `special:hotspot-internal` or just `hotspot-internal` as test
+    descriptor, and will only work on a debug JVM.
+
+  * Failure handler testing is run using `special:failure-handler` or just
+    `failure-handler` as test descriptor.
+
+  * Tests for the build system, including both makefiles and related
+    functionality, is run using `special:make` or just `make` as test
+    descriptor. This is equivalent to `special:make:all`.
+
+    A specific make test can be run by supplying it as argument, e.g.
+    `special:make:idea`. As a special syntax, this can also be expressed as
+    `make-idea`, which allows for command lines as `make test-make-idea`.
+
 ## Test results and summary
 
 At the end of the test run, a summary of all tests run will be presented. This
@@ -123,7 +150,7 @@
 classified as failed by the framework. This might indicate test framework
 error, timeout or other problems.
 
-In case of test failures, `make run-test` will exit with a non-zero exit value.
+In case of test failures, `make test` will exit with a non-zero exit value.
 
 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
 where TEST_ID is a path-safe conversion from the fully qualified test
--- a/make/Help.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/make/Help.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2012, 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2012, 2018, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -55,10 +55,10 @@
 	$(info $(_) make install           # Install the generated images locally)
 	$(info $(_) make reconfigure       # Rerun configure with the same arguments as last time)
 	$(info $(_) make help              # Give some help on using make)
-	$(info $(_) make test              # Run tests, default is all tests (see TEST below))
-	$(info $(_) make run-test-<test>   # Run test, e.g. run-test-tier1)
-	$(info $(_) make run-test TEST=<t> # Run test(s) given by TEST specification)
-	$(info $(_) make exploded-run-test TEST=<t> # Run test(s) on the exploded image instead of)
+	$(info $(_) make check             # Run basic testing (currently tier1))
+	$(info $(_) make test-<test>       # Run test, e.g. test-tier1)
+	$(info $(_) make test TEST=<t>     # Run test(s) given by TEST specification)
+	$(info $(_) make exploded-test TEST=<t> # Run test(s) on the exploded image instead of)
 	$(info $(_)                        # the full jdk image)
 	$(info )
 	$(info Targets for cleaning)
@@ -99,10 +99,12 @@
 	$(info $(_) TEST_JOBS=<n>          # Run <n> parallel test jobs)
 	$(info $(_) CONF_CHECK=<method>    # What to do if spec file is out of date)
 	$(info $(_)                        # method is 'auto', 'ignore' or 'fail' (default))
-	$(info $(_) make test TEST=<test>  # Only run the given test or tests, e.g.)
-	$(info $(_)                        # make test TEST="jdk_lang jdk_net")
-	$(info $(_) JTREG="OPT1=x;OPT2=y"  # Control the JTREG test harness for run-test)
-	$(info $(_) GTEST="OPT1=x;OPT2=y"  # Control the GTEST test harness for run-test)
+	$(info $(_) TEST="test1 ..."       # Use the given test descriptor(s) for testing, e.g.)
+	$(info $(_)                        # make test TEST="jdk_lang gtest:all")
+	$(info $(_) JTREG="OPT1=x;OPT2=y"  # Control the JTREG test harness)
+	$(info $(_) GTEST="OPT1=x;OPT2=y"  # Control the GTEST test harness)
+	$(info $(_) TEST_OPTS="OPT1=x;..." # Generic control of all test harnesses)
+	$(info $(_) TEST_VM_OPTS="ARG ..." # Same as setting TEST_OPTS to VM_OPTIONS="ARG ...")
 	$(info )
 	$(if $(all_confs), $(info Available configurations in $(build_dir):) $(foreach var,$(all_confs),$(info * $(var))),\
 	    $(info No configurations were found in $(build_dir).) $(info Run 'bash configure' to create a configuration.))
--- a/make/Main.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/make/Main.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -473,10 +473,10 @@
 #
 
 define DeclareRunTestRecipe
-  run-test-$1:
+  test-$1:
 	+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$1")
 
-  exploded-run-test-$1:
+  exploded-test-$1:
 	+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
 	    TEST="$1" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
 
@@ -484,8 +484,8 @@
 
 # ALL_NAMED_TESTS is defined in FindTests.gmk
 $(foreach t, $(ALL_NAMED_TESTS), $(eval $(call DeclareRunTestRecipe,$t)))
-ALL_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS))
-ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS))
+ALL_TEST_TARGETS := $(addprefix test-, $(ALL_NAMED_TESTS))
+ALL_EXPLODED_TEST_TARGETS := $(addprefix exploded-test-, $(ALL_NAMED_TESTS))
 
 ALL_TARGETS += $(ALL_TEST_TARGETS) $(ALL_EXPLODED_TEST_TARGETS)
 
@@ -520,13 +520,6 @@
 	+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f JtregGraalUnit.gmk \
 	     test-image-hotspot-jtreg-graal)
 
-run-test:
-	+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)")
-
-exploded-run-test:
-	+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
-	    TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
-
 ifeq ($(BUILD_GTEST), true)
   test-image-hotspot-gtest:
 	+($(CD) $(TOPDIR)/make/hotspot/test && $(MAKE) $(MAKE_ARGS) -f GtestImage.gmk)
@@ -541,11 +534,6 @@
 	+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
 	    -f BuildFailureHandler.gmk build)
 
-  # Runs the tests for the failure handler jtreg extension
-  test-failure-handler:
-	+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
-	    -f BuildFailureHandler.gmk test)
-
   # Copies the failure handler jtreg extension into the test image
   test-image-failure-handler:
 	+($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) \
@@ -556,40 +544,19 @@
     test-image-hotspot-jtreg-native build-test-jdk-jtreg-native \
     test-image-jdk-jtreg-native build-test-lib build-test-failure-handler \
     test-failure-handler test-image-failure-handler test-image-hotspot-gtest \
-    test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal \
-    run-test exploded-run-test
+    test-image-hotspot-jtreg-graal build-test-hotspot-jtreg-graal
 
 ################################################################################
 # Run tests
 
-# Run tests specified by $(TEST), or the default test set.
 test:
-	$(call RunTests, $(TEST), $(JDK_IMAGE_DIR))
+	+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test TEST="$(TEST)")
 
-test-hotspot-jtreg:
-	$(call RunTests, "hotspot_all", $(JDK_IMAGE_DIR))
+exploded-test:
+	+($(CD) $(TOPDIR)/make && $(MAKE) $(MAKE_ARGS) -f RunTests.gmk run-test \
+	    TEST="$(TEST)" JDK_IMAGE_DIR=$(JDK_OUTPUTDIR))
 
-test-hotspot-jtreg-native:
-	$(call RunTests, "hotspot_native_sanity", $(JDK_IMAGE_DIR))
-
-test-hotspot-internal:
-	$(call RunTests, "hotspot_internal", $(JDK_OUTPUTDIR))
-
-test-hotspot-gtest:
-	$(call RunTests, "hotspot_gtest", $(JDK_OUTPUTDIR))
-
-test-jdk-jtreg-native:
-	$(call RunTests, "jdk_native_sanity", $(JDK_IMAGE_DIR))
-
-test-make:
-	($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk $(TEST_TARGET))
-
-test-compile-commands:
-	($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f TestMake.gmk test-compile-commands)
-
-ALL_TARGETS += test test-hotspot-jtreg test-hotspot-jtreg-native \
-    test-hotspot-internal test-hotspot-gtest test-jdk-jtreg-native test-make \
-    test-compile-commands
+ALL_TARGETS += test exploded-test
 
 ################################################################################
 # Bundles
@@ -874,14 +841,18 @@
 
   docs-zip: docs-jdk
 
+  # Tests
   test: jdk-image test-image
 
-  run-test: jdk-image test-image
-  exploded-run-test: exploded-image test-image
+  exploded-test: exploded-image test-image
+
+  test-make: clean-test-make compile-commands
+
+  test-make-compile-commands: compile-commands
 
   # Declare dependency for all generated test targets
-  $(foreach t, $(ALL_TEST_TARGETS), $(eval $t: jdk-image test-image))
-  $(foreach t, $(ALL_EXPLODED_TEST_TARGETS), $(eval $t: exploded-image test-image))
+  $(foreach t, $(filter-out test-make%, $(ALL_TEST_TARGETS)), $(eval $t: jdk-image test-image))
+  $(foreach t, $(filter-out exploded-test-make%, $(ALL_EXPLODED_TEST_TARGETS)), $(eval $t: exploded-image test-image))
 
   create-buildjdk-copy: jdk.jlink-java java.base-gendata \
       $(addsuffix -java, $(INTERIM_IMAGE_MODULES))
@@ -890,16 +861,10 @@
 
   interim-image: $(INTERIM_JMOD_TARGETS)
 
-  test-make: clean-test-make
-
-  test-compile-commands: compile-commands
-
   build-test-lib: exploded-image-optimize
 
   build-test-failure-handler: interim-langtools
 
-  test-failure-handler: build-test-failure-handler
-
   test-image-failure-handler: build-test-failure-handler
 
   build-test-hotspot-jtreg-native: buildtools-jdk \
@@ -917,12 +882,6 @@
 
   test-image-hotspot-gtest: hotspot
 
-  test-hotspot-internal: exploded-image
-
-  test-hotspot-jtreg: jdk-image test-image
-
-  test-hotspot-gtest: exploded-image test-image-hotspot-gtest
-
   install: product-images
 
   product-bundles: product-images
@@ -1091,6 +1050,30 @@
 
 ALL_TARGETS += default jdk images docs bundles all
 
+# Aliases used for running tests.
+
+# Let "run-test" be an alias for "test"
+$(foreach t, $(ALL_NAMED_TESTS), $(eval run-test-$t: test-$t))
+$(foreach t, $(ALL_NAMED_TESTS), $(eval exploded-run-test-$t: exploded-test-$t))
+RUN_TEST_TARGETS := $(addprefix run-test-, $(ALL_NAMED_TESTS)) \
+    $(addprefix exploded-run-test-, $(ALL_NAMED_TESTS))
+
+run-test: test
+exploded-run-test: exploded-test
+
+# "make check" is a common idiom for running basic testing
+check: test-tier1
+
+# Keep some old names as aliases
+test-hotspot-jtreg: test-hotspot_all
+test-hotspot-jtreg-native: test-hotspot_native_sanity
+test-hotspot-gtest: exploded-test-gtest
+test-jdk-jtreg-native: test-jdk_native_sanity
+
+ALL_TARGETS += $(RUN_TEST_TARGETS) run-test exploded-run-test check \
+    test-hotspot-jtreg test-hotspot-jtreg-native test-hotspot-gtest \
+    test-jdk-jtreg-native
+
 ################################################################################
 ################################################################################
 #
--- a/make/MainSupport.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/make/MainSupport.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2011, 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2011, 2018, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -30,17 +30,6 @@
 ifndef _MAINSUPPORT_GMK
 _MAINSUPPORT_GMK := 1
 
-# Run the tests specified by $1, with PRODUCT_HOME specified by $2
-# JT_JAVA is picked up by the jtreg launcher and used to run Jtreg itself.
-define RunTests
-	($(CD) $(TOPDIR)/test && $(MAKE) $(MAKE_ARGS) -j1 -k MAKEFLAGS= \
-	    JT_HOME=$(JT_HOME) PRODUCT_HOME=$(strip $2) \
-	    TEST_IMAGE_DIR=$(TEST_IMAGE_DIR) \
-	    ALT_OUTPUTDIR=$(OUTPUTDIR) TEST_JOBS=$(TEST_JOBS) \
-	    JT_JAVA=$(BOOT_JDK) JIB_JAR=$(JIB_JAR) \
-	    JOBS=$(JOBS) $1) || true
-endef
-
 define CleanDocs
 	@$(PRINTF) "Cleaning docs ..."
 	@$(PRINTF) "\n" $(LOG_DEBUG)
--- a/make/RunTests.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/make/RunTests.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2016, 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2016, 2018, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -300,16 +300,32 @@
 
 # Helper function to determine if a test specification is a special test
 #
-# It is a special test if it is "special:" followed by a test name.
+# It is a special test if it is "special:" followed by a test name,
+# if it is "make:" or "make-" followed by a make test, or any of the special test names
+# as a single word.
 define ParseSpecialTestSelection
   $(if $(filter special:%, $1), \
     $1 \
+  ) \
+  $(if $(filter make%, $1), \
+    $(if $(filter make:%, $1), \
+      special:$(strip $1) \
+    ) \
+    $(if $(filter make-%, $1), \
+      special:$(patsubst make-%,make:%, $1) \
+    ) \
+    $(if $(filter make, $1), \
+      special:make:all \
+    )
+  ) \
+  $(if $(filter hotspot-internal failure-handler, $1), \
+    special:$(strip $1) \
   )
 endef
 
 ifeq ($(TEST), )
   $(info No test selection given in TEST!)
-  $(info Please use e.g. 'run-test TEST=tier1' or 'run-test-tier1')
+  $(info Please use e.g. 'make test TEST=tier1' or 'make test-tier1')
   $(info See doc/testing.[md|html] for help)
   $(error Cannot continue)
 endif
@@ -663,9 +679,13 @@
         $$(JDK_IMAGE_DIR)/bin/java -XX:+ExecuteInternalVMTests \
         -XX:+ShowMessageBoxOnError -version
   else ifeq ($$($1_TEST_NAME), failure-handler)
-    $1_TEST_COMMAND_LINE := \
-        ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \
-        BuildFailureHandler.gmk test)
+    ifeq ($(BUILD_FAILURE_HANDLER), true)
+      $1_TEST_COMMAND_LINE := \
+          ($(CD) $(TOPDIR)/make/test && $(MAKE) $(MAKE_ARGS) -f \
+          BuildFailureHandler.gmk test)
+    else
+      $$(error Cannot test failure handler if it is not built)
+    endif
   else ifeq ($$($1_TEST_NAME), make)
     $1_TEST_COMMAND_LINE := \
         ($(CD) $(TOPDIR)/test/make && $(MAKE) $(MAKE_ARGS) -f \
--- a/make/common/FindTests.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/make/common/FindTests.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2017, 2018, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -72,6 +72,15 @@
 # Add Gtest
 ALL_NAMED_TESTS += gtest
 
+# Find make test targets
+MAKE_TEST_TARGETS := $(shell $(MAKE) -s --no-print-directory $(MAKE_ARGS) \
+    SPEC=$(SPEC) -f $(TOPDIR)/test/make/TestMake.gmk print-targets)
+
+ALL_NAMED_TESTS += $(addprefix make-, $(MAKE_TEST_TARGETS))
+
+# Add special tests
+ALL_NAMED_TESTS += hotspot-internal failure-handler make
+
 ################################################################################
 
 endif # _FIND_TESTS_GMK
--- a/make/conf/jib-profiles.js	Wed Oct 31 08:09:45 2018 +0100
+++ b/make/conf/jib-profiles.js	Wed Oct 31 09:30:24 2018 +0100
@@ -759,7 +759,7 @@
                 testedProfile + ".test"
             ],
             src: "src.conf",
-            make_args: [ "run-test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ],
+            make_args: [ "test-prebuilt", "LOG_CMDLINES=true", "JTREG_VERBOSE=fail,error,time" ],
             environment: {
                 "BOOT_JDK": common.boot_jdk_home,
                 "JDK_IMAGE_DIR": input.get(testedProfile + ".jdk", "home_path"),
--- a/test/Makefile	Wed Oct 31 08:09:45 2018 +0100
+++ b/test/Makefile	Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2010, 2015, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2010, 2018, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -27,11 +27,16 @@
 # Makefile to run tests from multiple sibling directories
 #
 
+$(info WARNING: This way of running tests ("cd test && make") is deprecated)
+$(info Please use "make test TEST=..." instead. See doc/testing.md for details)
+
 # Macro to run a test target in a subdir
 define SUBDIR_TEST # subdirectory target
 if [ -d $1 ] ; then \
   if [ -r $1/Makefile ] ; then \
     $(MAKE) --no-print-directory -k -C $1 $2 ; \
+    echo 'WARNING: This way of running tests ("cd test && make") is deprecated' ; \
+    echo 'Please use "make test TEST=..." instead. See doc/testing.md for details' ; \
   else \
     echo "ERROR: File does not exist: $1/Makefile"; \
     exit 1; \
--- a/test/make/TestMake.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/test/make/TestMake.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2014, 2018, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # This code is free software; you can redistribute it and/or modify it
@@ -36,13 +36,17 @@
 copy-files:
 	+$(MAKE) -f TestCopyFiles.gmk $(TEST_SUBTARGET)
 
-test-idea:
+idea:
 	+$(MAKE) -f TestIdea.gmk $(TEST_SUBTARGET)
 
-test-compile-commands:
+compile-commands:
 	+$(MAKE) -f TestCompileCommands.gmk $(TEST_SUBTARGET)
 
+TARGETS += make-base java-compilation copy-files idea compile-commands
 
-all: make-base java-compilation copy-files test-idea
+all: $(TARGETS)
 
-.PHONY: default all make-base java-compilation copy-files test-idea test-compile-commands
+print-targets:
+	$(ECHO) "$(TARGETS)"
+
+.PHONY: default all $(TARGETS)
--- a/test/make/TestMakeBase.gmk	Wed Oct 31 08:09:45 2018 +0100
+++ b/test/make/TestMakeBase.gmk	Wed Oct 31 09:30:24 2018 +0100
@@ -344,7 +344,7 @@
 KWBASE := APA=banan;GURKA=tomat;COUNT=1%202%203%204%205;SUM=1+2+3+4+5;MANY_WORDS=I have the best words.
 
 $(eval $(call ParseKeywordVariable, KWBASE, \
-    KEYWORDS := APA GURKA SUM, \
+    SINGLE_KEYWORDS := APA GURKA SUM, \
     STRING_KEYWORDS := COUNT MANY_WORDS, \
 ))
 
@@ -377,7 +377,7 @@
 KWBASE_WEIRD := ;;APA=banan;;;GURKA=apelsin;APA=skansen;;
 
 $(eval $(call ParseKeywordVariable, KWBASE_WEIRD, \
-    KEYWORDS := APA GURKA SUM, \
+    SINGLE_KEYWORDS := APA GURKA SUM, \
     STRING_KEYWORDS := COUNT, \
 ))