Skip to content

Commit 6596629

Browse files
scottz64scottz64
committed
create regression test dirs
Fixes FAB-1369. Regression tests have been developed in multiple languages for blackbox testing of various components of hyperledger fabric. The intent is to soon place many tests scripts and supporting infrastructure under folder fabric/bddtests/regression/ to store them close to the fabric code with which they correlate. Next we will be constructing jenkins automation test suites and run them and post results for the community. Change-Id: Ie78ddd833e3c9792ff622847c9195703bbe10222 Signed-off-by: Scott Zwierzynski <[email protected]>
1 parent 0b98143 commit 6596629

File tree

9 files changed

+586
-0
lines changed

9 files changed

+586
-0
lines changed

‎bddtests/regression/README.md

+87
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
## regression
2+
Regression Test Suites scripts, and folders for: execution results for
3+
significant releases, and
4+
configuration scripts and supporting files in a variety of languages:
5+
6+
- [GO](https://github.com/hyperledger/fabric/bddtests/regression/go)
7+
language test scripts
8+
- [NODE](https://github.com/hyperledger/fabric/bddtests/regression/node)
9+
language test scripts
10+
Note: a flexible-use performance engine, which can be configured and used
11+
for a variety of tests, is located in the
12+
[node/performance](https://github.com/hyperledger/fabric/bddtests/regression/node/performance)
13+
folder. For examples how to use it, check there, or look for examples in the
14+
daily test suite and long-run test suite scripts.
15+
- [results](https://github.com/hyperledger/fabric/bddtests/regression/results)
16+
folder containing logfiles and results of running the Test Suites on
17+
significant releases, for reference
18+
19+
20+
### Continous Integration Setup
21+
Continuous Integration team will execute the *daily test suite* each day when a
22+
merge commit has been pushed to fabric repository.
23+
We have configured Jenkins in vLaunch to execute the Daily Test Suite:
24+
the jobs listed below are downstream jobs which are run after successfully
25+
executing the upstream job.
26+
After the build is successfully executed, Jenkins will post build results
27+
back to **rel-criteria-build** slack channel,
28+
and generate a test summary report, to be available for viewing for 30 days.
29+
For significant releases, the results will be stored
30+
in **bddtests/regression/results** folder.
31+
32+
33+
### Daily Test Suite - daily_test_suite.sh
34+
Expected total time duration is between 13 - 16 hours.
35+
36+
* Consensus Acceptance Tests (CAT) - using chaincode example02, GO, gRPC
37+
- Objective of CAT tests is to ensure the stability and resiliency of the
38+
BFT Batch design.
39+
* Large Networks Basic API And Consensus Tests - using chaincode example02,
40+
GO, gRPC
41+
- Objective of Basic gRPC API test is to ensure basic gRPC API functions
42+
are working as expected
43+
* Ledger Stress Tests (LST) for API and 20K runs with concurrency and
44+
1K payload - using chaincode addrecs, GO, gRPC
45+
* Speed Tests for measuring turnaround performance of the communication path
46+
and blockchain network processing - using chaincode addrecs, GO, gRPC
47+
* Concurrency Tests - using chaincode addrecs, GO, gRPC
48+
- Objective of Concurrency test is to ensure system is able to accept
49+
multiple threads concurrently for specified number of mins.
50+
* Complex Transactions Tests - using chaincode auction, node, gRPC
51+
- Objective of this test is to send complex transactions using
52+
auction chaincode
53+
* Performance Tests, random sized payloads - using chaincodes example02
54+
and auction, node, gRPC
55+
- CI performs below tests in Performance testing
56+
* Invoke on chaincode_example02 using 4 peers and
57+
4 Threads for 180 secs
58+
* Query on chaincode_example02 using 4 peers and
59+
4 Threads for 180 secs
60+
* Invoke on auction chaincode using 4 peers and
61+
1000 Tx for each of the 4 Threads
62+
* Query on auction chaincode using 4 peers and
63+
1000 Tx for each of the 4 Threads
64+
* Invoke on auction chaincode using 4 peers and 4 Threads.
65+
Each Invoke is followed by a Query on every Thread
66+
* Invoke on example02 chaincode using 4 peers and 4 Threads.
67+
Each Invoke is followed by a Query on every Thread
68+
69+
70+
### LongRun Test Suite - longrun_test_suite.sh
71+
Expected total time duration is 72 hours when executed from
72+
Jenkins automation scripts, which run several parallel jobs.
73+
74+
* Consensus peers restarts tests, 2 tests, each approximately 10 hours -
75+
using chaincode example02, GO, gRPC
76+
* Ledger Stress Test 1 Million transactions, approximately 36 hours -
77+
using chaincode addrecs, GO, gRPC
78+
* Ledger Stress Test, 72 hours - using chaincode addrecs, GO, gRPC
79+
- Send parallel Invoke requests on all 4 peers
80+
* Performance Tests, 72 hours, random sized payloads - using chaincode auction,
81+
node, gRPC (includes variable traffic, concurrency, complex transactions)
82+
- Invoke 1 Tx per sec on 1 peer for 72 hrs (1 Thread)
83+
84+
85+
### Full Regression Test Suite
86+
Execute both the Daily Test Suite AND the Long Run Test Suite.
87+

‎bddtests/regression/daily_test_suite.sh

+130
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
#!/bin/bash
2+
#
3+
# Copyright IBM Corp. 2016 All Rights Reserved.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# DESCRIPTION:
19+
# These are the tests in the Daily Test Suite.
20+
# To run these tests, user may clone the fabric, make code changes if
21+
# desired, build images, and run this script.
22+
23+
24+
# Setup
25+
26+
WORKDIR=`pwd`
27+
28+
29+
30+
31+
32+
# Note to test developers: Many tests in the Daily Test Suite are
33+
# specific tests that already contain the TEST OBJECTIVE and STEPS.
34+
# However, in some cases, such as running a flexible test engine to
35+
# perform different tests by using configuration parameters or
36+
# environment variables, it must be provided here.
37+
# For EACH of those test cases, before invoking the test, please
38+
# provide a comment block like this:
39+
40+
#-----------------------------------------------------------------------
41+
# TEST OBJECTIVE : basic network setup, deploy, invoke, query all peers
42+
# using gRPC interface, and exercise chaincode APIs
43+
# SETUP STEPS:
44+
# 1. Set environment variable: export TEST_NETWORK=LOCAL
45+
# TEST DETAILS (Optional):
46+
# 1. Setup local docker network with security with 4 peer nodes
47+
# 2. Deploy chaincode example02
48+
# 3. Send Invoke Requests on all peers using go routines
49+
# 4. Verify query results and chainheights match on all peers
50+
#-----------------------------------------------------------------------
51+
52+
53+
54+
55+
56+
########################################################################
57+
# Language API Functional Area
58+
# GO TDK gRPC Consensus Acceptance Tests
59+
########################################################################
60+
61+
cd $WORKDIR/go/tdk/CAT
62+
#UNDER CONSTRUCTION...
63+
#go run CAT_100_Startup_DIQ_API.go
64+
#go run CAT_101_BasicConsensus_S1_R1_S2_S1_R1_R2.go
65+
#go run CAT_102_S1_IQDQIQ.go
66+
67+
68+
69+
########################################################################
70+
# Language API Functional Area
71+
# GO TDK gRPC Transaction Speed Measurement
72+
########################################################################
73+
74+
cd $WORKDIR/go/tdk/ledgerstresstest
75+
#./run_speed_tests.sh
76+
77+
78+
79+
########################################################################
80+
# Language API Functional Area
81+
# GO TDK gRPC Ledger Stress Tests
82+
########################################################################
83+
84+
cd $WORKDIR/go/tdk/ledgerstresstest
85+
#go run BasicFuncExistingNetworkLST.go
86+
#go run LST_1client1peer20K.go
87+
#go run LST_2client1peer20K.go
88+
#go run LST_2client2peer20K.go
89+
#go run LST_4client1peer20K.go
90+
#go run LST_4client4peer20K.go
91+
92+
93+
94+
########################################################################
95+
# Language API Functional Area
96+
# GO TDK gRPC Concurrency/Ledger
97+
########################################################################
98+
99+
cd $WORKDIR/go/tdk/ledgerstresstest
100+
#go run conc4p1min1000Thrd1TxPerLoop_LOCAL.go
101+
#go run conc4p1min400Thrd_LOCAL.go
102+
103+
104+
105+
########################################################################
106+
# Language API Functional Area
107+
# Node.js gRPC Concurrency with Auction
108+
########################################################################
109+
110+
cd $WORKDIR/node/
111+
112+
113+
114+
########################################################################
115+
# Language API Functional Area
116+
# Node.js gRPC Complex Transactions with Auction
117+
########################################################################
118+
119+
cd $WORKDIR/node/
120+
121+
122+
123+
########################################################################
124+
# Language API Functional Area
125+
# Node.js gRPC Performance engine Baseline tests
126+
########################################################################
127+
128+
cd $WORKDIR/node/performance/
129+
130+

‎bddtests/regression/go/README.md

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## GO tests, scripts, and supporting files
2+
(For all scripts added to this directory, add information below to
3+
describe pre-requisites, setup steps, and usage examples.)
4+
5+
## SETUP
6+
7+
## USAGE
8+

‎bddtests/regression/go/tdk/CAT/README.md

+74
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
## Consensus Acceptance Testcases (CAT)
2+
Hyperledger Fabric Go language Test Development Kit (TDK)
3+
tests can be executed on local fabric in vagrant environment with docker
4+
containers.
5+
Additional details (the date the tests were run, commit image,
6+
test config parameters, the test steps, output summary and details, etc.)
7+
are available in the GO_TEST files in this folder.
8+
9+
### CAT test naming convention
10+
11+
The testnames themselves indicate the steps performed. For example:
12+
13+
```
14+
CAT_210_S2S1_IQ_R1_IQ.go Consensus Acceptance Test (CAT) suite, testcase number 210
15+
_S2S1 Stop Validating PEERs VP2 and VP1 at virtually the same time
16+
_IQ Send some Invoke requests to all running peers, and Query all to validate A/B/ChainHeight results
17+
_R1 Restart VP1
18+
_IQ Send some Invoke requests to all running peers, and Query all to validate A/B/ChainHeight results
19+
```
20+
21+
### Test Coverage
22+
23+
The objective of Consensus Acceptance Tests (CAT) is to ensure the
24+
stability and resiliency of the
25+
PBFT Batch design when Byzantine faults occur in a 4 peer network.
26+
Test areas coverage:
27+
28+
Stop 1 peer: the network continues to process deploys, invokes, and query
29+
transactions.
30+
Perform this operation on each peer in the fabric.
31+
While exactly 3 peers are running, their ledgers will remain in synch.
32+
(In a 4 peer network, 3 represents 2F+1, the minimum number required for
33+
consensus.)
34+
Restarting a 4th peer will cause it to join in the network operations again.
35+
Note: when queried after having been restarted, the ledger on extra
36+
peers ( additional nodes beyond 2(F+1) )
37+
may appear to lag for some time. It will catch up if another peer is
38+
stopped (leaving it as one of exactly 2F+1 participating peers), OR, with
39+
no further disruptions in the network,
40+
it could catch up after huge numbers of transaction batches are processed.
41+
42+
Stop 2 peers: the network should halt advancement, due to a lack of consensus.
43+
Restarting 1 or 2 peers will cause the network to resume processing
44+
transactions because enough nodes are available to reach consensus. This may
45+
include processing transactions that were received and queued by any running
46+
peers while consensus was halted.
47+
48+
Stop 3 peers: the network should halt advancement due to a lack of consensus.
49+
Restarting just one of the peers should not resume consensus.
50+
Restarting 2 or 3 peers should cause the network to resume consensus.
51+
52+
Deploys should be processed, or queued if appropriate, with any number of
53+
running peers.
54+
55+
56+
### RESULTS SUMMARY
57+
58+
```
59+
Consensus - Failed Testcases Opened Bugs Description
60+
61+
CAT_111_SnIQRnIQ_cycleDownLoop.go FAB-337 dup Tx
62+
CAT_303_S0S1S2_IQ_R0R1R2_IQ.go FAB-333 lost Tx after restart 3 peers together
63+
CAT_305_S1S2S3_IQ_R1R2R3_IQ.go FAB-333 lost Tx after restart 3 peers together
64+
CAT_407_S0S1S2_D_I_R0R1_IQ.go FAB-911 lost Tx from vp0 after stop vp0/vp1/vp2, deploy, restart vp0/vp1, invokes
65+
CAT_408_S0S1S2_D_I_R0R1R2_IQ.go FAB-335 deploy failed when restart 3 peers together
66+
CAT_409_S1S2S3_D_I_R1R2_IQ.go FAB-334/FAB-912 lost Tx after all peers after stop vp1/vp2/vp3, deploy, restart vp1/vp2, invokes
67+
CAT_410_S1S2S3_D_I_R1R2R3_IQ.go FAB-335 deploy failed when restart 3 peers together
68+
CAT_412_S0S1S2_D_I_R1R2_IQ.go FAB-334/FAB-912 lost Tx after all peers after stop vp1/vp2/vp3, deploy, restart vp1/vp2, invokes
69+
70+
CRT_502_StopAndRestart1or2_10Hrs.go FAB-331 acked Tx lost after stop/restart peers many times
71+
72+
CAT_201_S2S1_IQDIQ.go (using Pauses) FAB-336 no http response from vp0, while peers vp1,vp2 docker paused
73+
```
74+

0 commit comments

Comments
 (0)