You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
+ Renamed some files and testnames to meet the standardized
naming conventions, for a better user experience when
viewing test results on CI reports page for all the
tests in the daily and weekly test suite.
+ Added some scripts to facilitate running the
weekly test suite in groups in parallel.
+ Use Readme doc syntax strategy to use italics font format
for filenames and paths, and use bold font to highlight
other key phrases.
Change-Id: Icdecd4c5db74e10103daad2f2e4424cd5028a26a
Signed-off-by: Scott Zwierzynski <[email protected]>
Copy file name to clipboardexpand all lines: test/regression/daily/README.md
+24-24
Original file line number
Diff line number
Diff line change
@@ -1,61 +1,61 @@
1
1
# Daily Test Suite
2
2
3
-
This readme explains everything there is to know about our daily regression test suite. *Note 1*: This applies similarly for both the **test/regression/daily/** and **test/regression/weekly/** test suites. *Note 2*: The Release Criteria (**test/regression/release/**) test suite is a subset of all the Daily and Weekly tests.
3
+
This readme explains everything there is to know about our daily regression test suite. **Note 1**: This applies similarly for both the *test/regression/daily/* and *test/regression/weekly/* test suites. **Note 2**: The Release Criteria (*test/regression/release/*) test suite is a subset of all the Daily and Weekly tests.
4
4
5
5
- How to Run the Tests
6
6
- Where to View the Results produced by the daily automation tests
7
7
- Where to Find Existing Tests
8
8
- How to Add New Tests to the Automated Test Suite
9
-
* Why Test Output Format Must Be *xml* and How to Make It So
9
+
* Why Test Output Format Must Be **xml** and How to Make It So
10
10
* Alternative 1: Add a test using an existing tool and test driver script
11
11
* Alternative 2: Add a new test with a new tool and new test driver script
12
12
* How to Add a New Chaincode Test
13
13
14
14
## How to Run the Tests, and Where to View the Results
15
15
16
-
Everything starts with [runDailyTestSuite.sh](./runDailyTestSuite.sh), which invokes all test driver scripts, such as **test_pte.py** and **test_chaincodes.py**. Together, these driver scripts initiate all tests in the daily test suite. You can manually execute **runDailyTestSuite.sh** in its entirety, or, run one any one of the test driver scripts on the command line. Or, you may simply view the results generated daily by an automated Continuous Improvement (CI) tool which executes **runDailyTestSuite.sh**. Reports are displayed on the [Daily Test Suite Results Page](https://jenkins.hyperledger.org/view/Daily/job/fabric-daily-chaincode-tests-x86_64/test_results_analyzer). When you look at the reports; click the buttons in the **'See children'** column to see the results breakdown by component and by individual tests.
16
+
Everything starts with [runDailyTestSuite.sh](./runDailyTestSuite.sh), which invokes all test driver scripts, such as *systest_pte.py* and *chaincodes.py*. Together, these driver scripts initiate all tests in the daily test suite. You can manually execute *runDailyTestSuite.sh* in its entirety, or, run one any one of the test driver scripts on the command line. Or, you may simply view the results generated daily by an automated Continuous Improvement (CI) tool which executes *runDailyTestSuite.sh*. Reports are displayed on the [Daily Test Suite Results Page](https://jenkins.hyperledger.org/view/Daily/job/fabric-daily-chaincode-tests-x86_64/test_results_analyzer). When you look at the reports; click the buttons in the **'See children'** column to see the results breakdown by component and by individual tests.
17
17
18
18
#### Where to Find Existing Tests
19
19
20
-
Examine the driver scripts to find the individual tests, which are actually stored in several locations under **/path/to/fabric/test/**. Some tests are located in test suite subdirectories such as
20
+
Examine the driver scripts to find the individual tests, which are actually stored in several locations under */path/to/fabric/test/*. Some tests are located in test suite subdirectories such as
21
21
22
-
-**test/regression/daily/chaincodeTests/**
22
+
-*test/regression/daily/chaincodeTests/*
23
23
24
24
whereas other tests are located in the tools directories themselves, such as
-*test/tools/PTE/* - Performance Traffic Engine **(PTE)** tool and tests
28
+
-*test/tools/OTE/* - Orderer Traffic Engine **(OTE)** tool and tests
29
29
30
30
Each testcase title should provide the test objective and a Jira FAB issue which can be referenced for more information. Test steps and specific details can be found in the summary comments of the test scripts themselves. Additional information can be found in the README files associated with the various test directories.
31
31
32
32
## How to Add New Tests to the Automated Test Suite
33
33
34
-
We love contributors! Anyone may add a new test to an existing test driver script, or even create a new tool and new test driver script. The steps for both scenarios are provided further below as *Alternative 1* and *Alternative 2*. First, a few things to note:
34
+
We love contributors! Anyone may add a new test to an existing test driver script, or even create a new tool and new test driver script. The steps for both scenarios are provided further below as **Alternative 1** and **Alternative 2**. First, a few things to note:
35
35
36
36
- Before linking a test case into the CI automation tests, please merge your (tool and) testcase into gerrit, and create a Jira task, as follows:
37
37
38
-
1. First merge your tool and tests to gerrit in appropriate folders under **/path/to/fabric/test/**.
38
+
1. First merge your tool and tests to gerrit in appropriate folders under */path/to/fabric/test/*.
39
39
1. Of course, all tests must pass before being submitted. We do not want to see any false positives for test case failures.
40
40
1. To integrate your new tests into the CI automation test suite, create a new Jira task FAB-nnnn for each testcase, and use 'relates-to' to link it to epic FAB-3770.
41
-
1. You will this new Jira task to submit a changeset to gerrit, to invoke your testcase from a driver script similar to **/path/to/fabric/test/regression/daily/test_example.py**. In the comments of the gerrit merge request submission, include the
41
+
1. You will this new Jira task to submit a changeset to gerrit, to invoke your testcase from a driver script similar to */path/to/fabric/test/regression/daily/Example.py*. In the comments of the gerrit merge request submission, include the
42
42
- Jira task FAB-nnnn
43
43
- the testcase title and objective
44
44
- copy and fill in the template from Jira epic FAB-3770
45
-
1. Follow all the steps below in either *Alternative*, and then the test will be executed automatically as part of the next running of the CI daily test suite. The results will show up on the daily test suite display board - which can be viewed by following the link at the top of this page.
45
+
1. Follow all the steps below in either **Alternative**, and then the test will be executed automatically as part of the next running of the CI daily test suite. The results will show up on the daily test suite display board - which can be viewed by following the link at the top of this page.
46
46
47
-
#### Why Test Output Format Must Be *xml* and How to Make It So
47
+
#### Why Test Output Format Must Be **xml** and How to Make It So
48
48
49
-
The Continuous Improvement (CI) team utilizes a Jenkins job to execute the full test suite, **runDailyTestSuite.sh**. The CI job consumes xml output files, creates reports, and displays them. *Note: When adding new scripts that generate new xml files, if you do not see the results displayed correctly, please contact us on [Rocket.Chat channel #fabric-ci](https://chat.hyperledger.org).* For this reason, we execute tests in one of the following ways:
49
+
The Continuous Improvement (CI) team utilizes a Jenkins job to execute the full test suite, *runDailyTestSuite.sh*. The CI job consumes xml output files, creates reports, and displays them. **Note:** When adding new scripts that generate new xml files, if you do not see the results displayed correctly, please contact us on [Rocket.Chat channel #fabric-ci](https://chat.hyperledger.org). For this reason, we execute tests in one of the following ways:
50
50
51
-
1. Invoke the individual testcase from within a test driver script in **regression/daily/**. There are many examples here, such as **test_example.py** and **test_pte.py**. These test driver scripts are basically wrappers written in python, which makes it easy to produce the desired junitxml output format required for displaying reports. This method is useful for almost any test language, including bash, tool binaries, and more. More details are provided below explaining how to call testcases from within a test driver script. Here we show how simple it is to execute the test driver and all the testcases within it. *Note: File 'example_results.xml' will be created, containing the test output.*
51
+
1. Invoke the individual testcase from within a test driver script in *regression/daily/*. There are many examples here, such as *Example.py* and *systest_pte.py*. These test driver scripts are basically wrappers written in python, which makes it easy to produce the desired junitxml output format required for displaying reports. This method is useful for almost any test language, including bash, tool binaries, and more. More details are provided below explaining how to call testcases from within a test driver script. Here we show how simple it is to execute the test driver and all the testcases within it. **Note:** File *results_sample.xml* will be created, containing the sample testcases output.
1. Execute 'go test', and pipe the output through tool github.com/jstemmer/go-junit-report to convert to xml. *Note: In the example shown, file 'results.xml' will be created with the test output.*
58
+
1. Execute 'go test', and pipe the output through tool github.com/jstemmer/go-junit-report to convert to xml. **Note:** In the example shown, file 'results.xml' will be created with the test output.
59
59
60
60
```
61
61
cd /path/to/fabric/test/tools/OTE
@@ -67,14 +67,14 @@ The Continuous Improvement (CI) team utilizes a Jenkins job to execute the full
67
67
68
68
### Alternative 1: Add a test using an existing tool and test driver script
69
69
70
-
To add another test using an existing tool (such as **PTE**), simply add a test inside the existing test driver (such as **test_pte.py**). It is as simple as copying a block of ten lines and modify these things:
70
+
To add another test using an existing tool (such as **PTE**), simply add a test inside the existing test driver (such as *systest_pte.py*). It is as simple as copying a block of ten lines and modify these things:
71
71
72
72
1. Insert the testcase in the correct test component class and edit the test name
73
73
1. Edit the testcase description
74
74
1. Edit the specified command and arguments to be executed
75
75
1. Edit the asserted test result to be matched
76
76
77
-
Refer to **test_example.py** for a model to clone and get started quickly. The testcases should use the format shown in this example:
77
+
Refer to *Example.py* for a model to clone and get started quickly. The testcases should use the format shown in this example:
78
78
79
79
```
80
80
def test_FAB9876_1K_Payload(self):
@@ -94,9 +94,9 @@ Refer to **test_example.py** for a model to clone and get started quickly. The t
94
94
95
95
Adding a new test with a new tool involves a few more steps.
96
96
97
-
1. Create and merge a new tool, for example, **/path/to/fabric/test/tools/NewTool/newTool.sh**
98
-
1. Create a new test driver script such as **/path/to/fabric/test/regression/daily/test_newTool.py**. Model it after others like **test_example.py**, found under driver scripts under **/path/to/test/regression/daily/** and **test/regression/weekly/**. Note: the filename must start with 'test_'.
99
-
1. Add your new testcases to **test_newTool.py**. The testcases should use the following format. Refer also to the steps described in Alternative 1, above.
97
+
1. Create and merge a new tool, for example, */path/to/fabric/test/tools/NewTool/newTool.sh*
98
+
1. Create a new test driver script such as */path/to/fabric/test/regression/daily/newTool.py*. Model it after others like *Example.py*, found under driver scripts under */path/to/test/regression/daily/* and *test/regression/weekly/*.
99
+
1. Add your new testcases to *newTool.py*. The testcases should use the following format. Refer also to the steps described in Alternative 1, above.
100
100
101
101
```
102
102
class <component_feature>(unittest.TestCase):
@@ -109,10 +109,10 @@ Adding a new test with a new tool involves a few more steps.
109
109
self.assertIn("<string from stdout of newTool that indicates PASS>", result)
110
110
```
111
111
112
-
1. Edit **/path/to/test/regression/daily/runDailyTestSuite.sh** to run the new testcases. Add a new line, or append your new test driver scriptname **test_newTool.py** to an existing line:
112
+
1. Edit */path/to/test/regression/daily/runDailyTestSuite.sh* to run the new testcases. Add a new line, or append your new test driver scriptname *newTool.py* to an existing line:
0 commit comments