annotate doc/testing.md @ 51161:afe2cecf8867

Added tag jdk-11+23 for changeset 1edcf36fe15f
author prr
date Thu, 19 Jul 2018 09:42:15 -0700
parents 1582de22e3a1
children 9341b077bd55
rev   line source
mr@50885 1 % Testing the JDK
ihse@44511 2
ihse@44511 3 ## Using the run-test framework
ihse@44511 4
ihse@44511 5 This new way of running tests is developer-centric. It assumes that you have
mr@50885 6 built a JDK locally and want to test it. Running common test targets is simple,
ihse@44511 7 and more complex ad-hoc combination of tests is possible. The user interface is
ihse@44511 8 forgiving, and clearly report errors it cannot resolve.
ihse@44511 9
erikj@45266 10 The main target "run-test" uses the jdk-image as the tested product. There is
erikj@45266 11 also an alternate target "exploded-run-test" that uses the exploded image
erikj@45266 12 instead. Not all tests will run successfully on the exploded image, but using
erikj@45266 13 this target can greatly improve rebuild times for certain workflows.
erikj@45266 14
ihse@44511 15 Some example command-lines:
ihse@44511 16
ihse@44511 17 $ make run-test-tier1
ihse@44511 18 $ make run-test-jdk_lang JTREG="JOBS=8"
ihse@44511 19 $ make run-test TEST=jdk_lang
ihse@44511 20 $ make run-test-only TEST="gtest:LogTagSet gtest:LogTagSetDescriptions" GTEST="REPEAT=-1"
ihse@50267 21 $ make run-test TEST="hotspot:hotspot_gc" JTREG="JOBS=1;TIMEOUT=8;VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"
ihse@50267 22 $ make run-test TEST="jtreg:test/hotspot:hotspot_gc test/hotspot/jtreg/native_sanity/JniVersion.java"
ihse@50267 23 $ make exploded-run-test TEST=tier2
ihse@44511 24
ihse@45618 25 ### Configuration
ihse@45618 26
ihse@45618 27 To be able to run JTReg tests, `configure` needs to know where to find the
ihse@45618 28 JTReg test framework. If it is not picked up automatically by configure, use
ihse@45618 29 the `--with-jtreg=<path to jtreg home>` option to point to the JTReg framework.
ihse@45618 30 Note that this option should point to the JTReg home, i.e. the top directory,
ihse@45618 31 containing `lib/jtreg.jar` etc. (An alternative is to set the `JT_HOME`
ihse@45618 32 environment variable to point to the JTReg home before running `configure`.)
ihse@45618 33
ihse@44511 34 ## Test selection
ihse@44511 35
ihse@44511 36 All functionality is available using the run-test make target. In this use
ihse@44511 37 case, the test or tests to be executed is controlled using the `TEST` variable.
ihse@44511 38 To speed up subsequent test runs with no source code changes, run-test-only can
ihse@44511 39 be used instead, which do not depend on the source and test image build.
ihse@44511 40
ihse@44511 41 For some common top-level tests, direct make targets have been generated. This
ihse@45618 42 includes all JTReg test groups, the hotspot gtest, and custom tests (if
ihse@44511 43 present). This means that `make run-test-tier1` is equivalent to `make run-test
ihse@44511 44 TEST="tier1"`, but the latter is more tab-completion friendly. For more complex
ihse@44511 45 test runs, the `run-test TEST="x"` solution needs to be used.
ihse@44511 46
ihse@44511 47 The test specifications given in `TEST` is parsed into fully qualified test
ihse@44511 48 descriptors, which clearly and unambigously show which tests will be run. As an
ihse@50267 49 example, `:tier1` will expand to `jtreg:$(TOPDIR)/test/hotspot/jtreg:tier1
ihse@50267 50 jtreg:$(TOPDIR)/test/jdk:tier1 jtreg:$(TOPDIR)/test/langtools:tier1
ihse@50267 51 jtreg:$(TOPDIR)/test/nashorn:tier1 jtreg:$(TOPDIR)/test/jaxp:tier1`. You can
ihse@50267 52 always submit a list of fully qualified test descriptors in the `TEST` variable
ihse@50267 53 if you want to shortcut the parser.
ihse@44511 54
ihse@45618 55 ### JTReg
ihse@44511 56
ihse@50267 57 JTReg tests can be selected either by picking a JTReg test group, or a selection
ihse@50267 58 of files or directories containing JTReg tests.
ihse@50267 59
ihse@45618 60 JTReg test groups can be specified either without a test root, e.g. `:tier1`
ihse@50267 61 (or `tier1`, the initial colon is optional), or with, e.g. `hotspot:tier1`,
ihse@50267 62 `test/jdk:jdk_util` or `$(TOPDIR)/test/hotspot/jtreg:hotspot_all`. The test
ihse@50267 63 root can be specified either as an absolute path, or a path relative to the
mr@50885 64 JDK top directory, or the `test` directory. For simplicity, the hotspot
ihse@50267 65 JTReg test root, which really is `hotspot/jtreg` can be abbreviated as
ihse@50267 66 just `hotspot`.
ihse@44511 67
ihse@50267 68 When specified without a test root, all matching groups from all test roots
ihse@44511 69 will be added. Otherwise, only the group from the specified test root will be
ihse@44511 70 added.
ihse@44511 71
ihse@45618 72 Individual JTReg tests or directories containing JTReg tests can also be
ihse@50267 73 specified, like `test/hotspot/jtreg/native_sanity/JniVersion.java` or
ihse@50267 74 `hotspot/jtreg/native_sanity`. Just like for test root selection, you can
ihse@50267 75 either specify an absolute path (which can even point to JTReg tests outside
mr@50885 76 the source tree), or a path relative to either the JDK top directory or the
ihse@50267 77 `test` directory. `hotspot` can be used as an alias for `hotspot/jtreg` here as
ihse@50267 78 well.
ihse@44511 79
ihse@44511 80 As long as the test groups or test paths can be uniquely resolved, you do not
ihse@44511 81 need to enter the `jtreg:` prefix. If this is not possible, or if you want to
ihse@44511 82 use a fully qualified test descriptor, add `jtreg:`, e.g.
ihse@50267 83 `jtreg:test/hotspot/jtreg/native_sanity`.
ihse@44511 84
ihse@44511 85 ### Gtest
ihse@44511 86
ihse@44511 87 Since the Hotspot Gtest suite is so quick, the default is to run all tests.
ihse@44511 88 This is specified by just `gtest`, or as a fully qualified test descriptor
ihse@44511 89 `gtest:all`.
ihse@44511 90
ihse@44511 91 If you want, you can single out an individual test or a group of tests, for
ihse@44511 92 instance `gtest:LogDecorations` or `gtest:LogDecorations.level_test_vm`. This
ihse@44511 93 can be particularly useful if you want to run a shaky test repeatedly.
ihse@44511 94
ihse@48058 95 For Gtest, there is a separate test suite for each JVM variant. The JVM variant
ihse@48058 96 is defined by adding `/<variant>` to the test descriptor, e.g.
ihse@48058 97 `gtest:Log/client`. If you specify no variant, gtest will run once for each JVM
ihse@48058 98 variant present (e.g. server, client). So if you only have the server JVM
ihse@48058 99 present, then `gtest:all` will be equivalent to `gtest:all/server`.
ihse@48058 100
ihse@44511 101 ## Test results and summary
ihse@44511 102
ihse@44511 103 At the end of the test run, a summary of all tests run will be presented. This
ihse@44511 104 will have a consistent look, regardless of what test suites were used. This is
ihse@44511 105 a sample summary:
ihse@44511 106
ihse@44511 107 ==============================
ihse@44511 108 Test summary
ihse@44511 109 ==============================
ihse@44511 110 TEST TOTAL PASS FAIL ERROR
ihse@44511 111 >> jtreg:jdk/test:tier1 1867 1865 2 0 <<
ihse@44511 112 jtreg:langtools/test:tier1 4711 4711 0 0
ihse@44511 113 jtreg:nashorn/test:tier1 133 133 0 0
ihse@44511 114 ==============================
ihse@44511 115 TEST FAILURE
ihse@44511 116
ihse@44511 117 Tests where the number of TOTAL tests does not equal the number of PASSed tests
ihse@44511 118 will be considered a test failure. These are marked with the `>> ... <<` marker
ihse@44511 119 for easy identification.
ihse@44511 120
ihse@44511 121 The classification of non-passed tests differs a bit between test suites. In
ihse@44511 122 the summary, ERROR is used as a catch-all for tests that neither passed nor are
ihse@44511 123 classified as failed by the framework. This might indicate test framework
ihse@44511 124 error, timeout or other problems.
ihse@44511 125
ihse@44511 126 In case of test failures, `make run-test` will exit with a non-zero exit value.
ihse@44511 127
ihse@45618 128 All tests have their result stored in `build/$BUILD/test-results/$TEST_ID`,
ihse@44511 129 where TEST_ID is a path-safe conversion from the fully qualified test
ihse@44511 130 descriptor, e.g. for `jtreg:jdk/test:tier1` the TEST_ID is
ihse@44511 131 `jtreg_jdk_test_tier1`. This path is also printed in the log at the end of the
ihse@44511 132 test run.
ihse@44511 133
ihse@44511 134 Additional work data is stored in `build/$BUILD/test-support/$TEST_ID`. For
ihse@44511 135 some frameworks, this directory might contain information that is useful in
ihse@44511 136 determining the cause of a failed test.
ihse@44511 137
ihse@44511 138 ## Test suite control
ihse@44511 139
ihse@44511 140 It is possible to control various aspects of the test suites using make control
ihse@44511 141 variables.
ihse@44511 142
ihse@44511 143 These variables use a keyword=value approach to allow multiple values to be
ihse@45618 144 set. So, for instance, `JTREG="JOBS=1;TIMEOUT=8"` will set the JTReg
ihse@44511 145 concurrency level to 1 and the timeout factor to 8. This is equivalent to
ihse@44511 146 setting `JTREG_JOBS=1 JTREG_TIMEOUT=8`, but using the keyword format means that
ihse@44511 147 the `JTREG` variable is parsed and verified for correctness, so
ihse@44511 148 `JTREG="TMIEOUT=8"` would give an error, while `JTREG_TMIEOUT=8` would just
ihse@44511 149 pass unnoticed.
ihse@44511 150
ihse@44511 151 To separate multiple keyword=value pairs, use `;` (semicolon). Since the shell
ihse@44511 152 normally eats `;`, the recommended usage is to write the assignment inside
ihse@44511 153 qoutes, e.g. `JTREG="...;..."`. This will also make sure spaces are preserved,
iignatyev@49293 154 as in `JTREG="VM_OPTIONS=-XshowSettings -Xlog:gc+ref=debug"`.
ihse@44511 155
ihse@44511 156 (Other ways are possible, e.g. using backslash: `JTREG=JOBS=1\;TIMEOUT=8`.
ihse@44511 157 Also, as a special technique, the string `%20` will be replaced with space for
iignatyev@49293 158 certain options, e.g. `JTREG=VM_OPTIONS=-XshowSettings%20-Xlog:gc+ref=debug`.
ihse@44511 159 This can be useful if you have layers of scripts and have trouble getting
ihse@44511 160 proper quoting of command line arguments through.)
ihse@44511 161
ihse@44511 162 As far as possible, the names of the keywords have been standardized between
ihse@44511 163 test suites.
ihse@44511 164
ihse@45618 165 ### JTReg keywords
ihse@44511 166
ihse@44511 167 #### JOBS
ihse@44511 168 The test concurrency (`-concurrency`).
ihse@44511 169
ihse@44511 170 Defaults to TEST_JOBS (if set by `--with-test-jobs=`), otherwise it defaults to
ihse@44511 171 JOBS, except for Hotspot, where the default is *number of CPU cores/2*, but
ihse@44511 172 never more than 12.
ihse@44511 173
ihse@44511 174 #### TIMEOUT
ihse@44511 175 The timeout factor (`-timeoutFactor`).
ihse@44511 176
ihse@44511 177 Defaults to 4.
ihse@44511 178
ihse@44511 179 #### TEST_MODE
ihse@44511 180 The test mode (`-agentvm`, `-samevm` or `-othervm`).
ihse@44511 181
ihse@44511 182 Defaults to `-agentvm`.
ihse@44511 183
ihse@44511 184 #### ASSERT
ihse@44511 185 Enable asserts (`-ea -esa`, or none).
ihse@44511 186
ihse@44511 187 Set to `true` or `false`. If true, adds `-ea -esa`. Defaults to true, except
ihse@44511 188 for hotspot.
ihse@44511 189
ihse@44511 190 #### VERBOSE
ihse@44511 191 The verbosity level (`-verbose`).
ihse@44511 192
ihse@44511 193 Defaults to `fail,error,summary`.
ihse@44511 194
ihse@44511 195 #### RETAIN
ihse@44511 196 What test data to retain (`-retain`).
ihse@44511 197
ihse@44511 198 Defaults to `fail,error`.
ihse@44511 199
ihse@44511 200 #### MAX_MEM
ihse@44511 201 Limit memory consumption (`-Xmx` and `-vmoption:-Xmx`, or none).
ihse@44511 202
ihse@45618 203 Limit memory consumption for JTReg test framework and VM under test. Set to 0
ihse@44511 204 to disable the limits.
ihse@44511 205
ihse@44511 206 Defaults to 512m, except for hotspot, where it defaults to 0 (no limit).
ihse@44511 207
ihse@44511 208 #### OPTIONS
ihse@45618 209 Additional options to the JTReg test framework.
ihse@44511 210
ihse@45618 211 Use `JTREG="OPTIONS=--help all"` to see all available JTReg options.
ihse@44511 212
ihse@44511 213 #### JAVA_OPTIONS
ihse@45618 214 Additional Java options to JTReg (`-javaoption`).
ihse@44511 215
ihse@44511 216 #### VM_OPTIONS
ihse@45618 217 Additional VM options to JTReg (`-vmoption`).
ihse@44511 218
ihse@44511 219 ### Gtest keywords
ihse@44511 220
ihse@44511 221 #### REPEAT
ihse@44511 222 The number of times to repeat the tests (`--gtest_repeat`).
ihse@44511 223
ihse@44511 224 Default is 1. Set to -1 to repeat indefinitely. This can be especially useful
ihse@44511 225 combined with `OPTIONS=--gtest_break_on_failure` to reproduce an intermittent
ihse@44511 226 problem.
ihse@44511 227
ihse@44511 228 #### OPTIONS
ihse@44511 229 Additional options to the Gtest test framework.
ihse@44511 230
ihse@44511 231 Use `GTEST="OPTIONS=--help"` to see all available Gtest options.
ihse@44511 232
ihse@44511 233 ---
ihse@44983 234 # Override some definitions in the global css file that are not optimal for
ihse@44983 235 # this document.
ihse@44511 236 header-includes:
ihse@44511 237 - '<style type="text/css">pre, code, tt { color: #1d6ae5; }</style>'
ihse@44511 238 ---