Move Docker-Compose files into their own folder

Move the BDD Docker compose files into a new sub folder so they are all
in one place and do not clog up the root of the bddtests/ folder.
While altering logic to determine the container names I also refactored
the Docker-Compose parsing code to make it more clear as to what it was
doing (parseComposeOutput in bdd_compose_util.py).

I needed to add a dummy compose-defaults.yml file so that CI would
successfully pass. When this change makes it into all branches then it
can be removed and the CI scripts updated.

I also changed the permissions on a couple of the bdd files from 755 to
644.

Signed-off-by: Julian Carrivick <[email protected]>
Change-Id: Iefd1007e79ee3827d39ed9b0bdec327f30e0c39f

Author: juliancarrivick-ibm

bddtests/bdd-docker/README.md

@@ -0,0 +1,20 @@
+# BDD Docker Compose Files
+These files are utilized by the BDD tests to provide standardized environments
+for running tests cases against. They also conveniently demonstrate some of the
+different configurations that the peers can be run in.
+
+For convenience, all the configurations inherit from `compose-defaults.yml`.
+
+Two important notes:
+
+ 1. If a shell script is used as the docker comment to run (say, to add
+ additional logic before starting a peer) then the shell script and the command
+ should be invoked with an exec statement so that the signals that docker sends
+ to the container get through to the process. Otherwise there will be a 10
+ second when stopping a container before docker sends a SIGKILL.
+ See the command used for the peer in `compose-defaults.yml` for an example
+
+ 2. The name of peers should adhere to the convention of `vp[0-9]+` as this is
+ the method as to which peers (as opposed to the membersrvc) are detected during
+ the BDD runs. Peers are subjected to additional constraints before being
+ considered 'ready'.

bddtests/bdd-docker/compose-defaults.yml

@@ -0,0 +1,25 @@
+vp:
+  image: hyperledger/fabric-peer
+  environment:
+    - CORE_PEER_ADDRESSAUTODETECT=true
+    - CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock
+    # TODO:  This is currently required due to BUG in variant logic based upon log level.
+    - CORE_LOGGING_LEVEL=DEBUG
+  # Script will wait until membersrvc is up (if it exists) before starting
+  # $$GOPATH (double dollar) required to prevent docker-compose doing its own
+  # substitution before the value gets to the container
+  command: sh -c "exec $$GOPATH/src/github.com/hyperledger/fabric/bddtests/scripts/start-peer.sh"
+  # Ideally we'd only mount /var/run/docker.sock but v1.5.2 of docker-compose
+  # does not have to capability to mount individual files. Hence we mount the
+  # entire folder in specific folder and specify it explicitly above.
+  # This issue seems to be sorted in docker-compose 1.8.0 however that requires
+  # Docker 1.10 and CI isn't at that version yet.
+  volumes:
+    - /var/run/:/host/var/run/
+
+  # Use these options if coverage desired for peers
+  #image: hyperledger/fabric-peer-coverage
+  #command: ./peer.test --test.coverprofile=coverage.cov node start
+membersrvc:
+   image: hyperledger/fabric-membersrvc
+   command: membersrvc

bddtests/docker-compose-1-devmode.yml bddtests/bdd-docker/docker-compose-1-devmode.yml

@@ -0,0 +1,3 @@
+vp3:
+  environment:
+    - CORE_PBFT_GENERAL_BYZANTINE=true

bddtests/docker-compose-1-empty.yml bddtests/bdd-docker/docker-compose-1-empty.yml

@@ -1,18 +1,4 @@
-vp:
-  image: hyperledger/fabric-peer
-  environment:
-    - CORE_PEER_ADDRESSAUTODETECT=true
-    - CORE_VM_ENDPOINT=http://172.17.0.1:2375
-    # TODO:  This is currently required due to BUG in variant logic based upon log level.
-    - CORE_LOGGING_LEVEL=DEBUG
-  # Script will wait until membersrvc is up (if it exists) before starting
-  # $$GOPATH (double dollar) required to prevent docker-compose doing its own
-  # substitution before the value gets to the container
-  command: sh -c "exec $$GOPATH/src/github.com/hyperledger/fabric/bddtests/scripts/start-peer.sh"
-
-  # Use these options if coverage desired for peers
-  #image: hyperledger/fabric-peer-coverage
-  #command: ./peer.test --test.coverprofile=coverage.cov node start
-membersrvc:
-   image: hyperledger/fabric-membersrvc
-   command: membersrvc
+- |
+  This file is a simple dummy to satisfy the CI scripts which make modifications
+  to this file. This file now lives in bddtests/bdd-docker. When the CI scripts
+  are modified to remove that logic, this file can be removed.

bddtests/docker-compose-1-exp.yml bddtests/bdd-docker/docker-compose-1-exp.yml

@@ -3,20 +3,13 @@
 import glob
 
 from steps.bdd_test_util import cli_call, bdd_log
+from steps.bdd_compose_util import getDockerComposeFileArgsFromYamlFile
 
 from steps.coverage import saveCoverageFiles, createCoverageAggregate
 
 def coverageEnabled(context):
     return context.config.userdata.get("coverage", "false") == "true"
 
-
-def getDockerComposeFileArgsFromYamlFile(compose_yaml):
-    parts = compose_yaml.split()
-    args = []
-    for part in parts:
-        args = args + ["-f"] + [part]
-    return args
-
 def after_scenario(context, scenario):
     get_logs = context.config.userdata.get("logs", "N")
     if get_logs.lower() == "force" or (scenario.status == "failed" and get_logs.lower() == "y" and "compose_containers" in context):

bddtests/docker-compose-1-profiling.yml bddtests/bdd-docker/docker-compose-1-profiling.yml

@@ -43,50 +43,29 @@ def __str__(self):
     def __repr__(self):
         return self.__str__()
 
+DOCKER_COMPOSE_FOLDER = "bdd-docker"
+
 def getDockerComposeFileArgsFromYamlFile(compose_yaml):
     parts = compose_yaml.split()
     args = []
     for part in parts:
+        part = "{}/{}".format(DOCKER_COMPOSE_FOLDER, part)
         args = args + ["-f"] + [part]
     return args
 
 def parseComposeOutput(context):
     """Parses the compose output results and set appropriate values into context.  Merges existing with newly composed."""
-    # Use the prefix to get the container name
-    containerNamePrefix = os.path.basename(os.getcwd()) + "_"
-    containerNames = []
-    for l in context.compose_error.splitlines():
-        tokens = l.split()
-        bdd_log(tokens)
-        if 1 < len(tokens):
-            thisContainer = tokens[1]
-            if containerNamePrefix not in thisContainer:
-               thisContainer = containerNamePrefix + thisContainer + "_1"
-            if thisContainer not in containerNames:
-               containerNames.append(thisContainer)
+    containerNames = getContainerNamesFromContext(context)
 
     bdd_log("Containers started: ")
     bdd_log(containerNames)
     # Now get the Network Address for each name, and set the ContainerData onto the context.
     containerDataList = []
     for containerName in containerNames:
-        output, error, returncode = \
-            cli_call(["docker", "inspect", "--format",  "{{ .NetworkSettings.IPAddress }}", containerName], expect_success=True)
-        bdd_log("container {0} has address = {1}".format(containerName, output.splitlines()[0]))
-        ipAddress = output.splitlines()[0]
+        ipAddress = getIpFromContainerName(containerName)
+        env = getEnvironmentVariablesFromContainerName(containerName)
+        dockerComposeService = getDockerComposeServiceForContainer(containerName)
 
-        # Get the environment array
-        output, error, returncode = \
-            cli_call(["docker", "inspect", "--format",  "{{ .Config.Env }}", containerName], expect_success=True)
-        env = output.splitlines()[0][1:-1].split()
-
-        # Get the Labels to access the com.docker.compose.service value
-        output, error, returncode = \
-            cli_call(["docker", "inspect", "--format",  "{{ .Config.Labels }}", containerName], expect_success=True)
-        labels = output.splitlines()[0][4:-1].split()
-        dockerComposeService = [composeService[27:] for composeService in labels if composeService.startswith("com.docker.compose.service:")][0]
-        bdd_log("dockerComposeService = {0}".format(dockerComposeService))
-        bdd_log("container {0} has env = {1}".format(containerName, env))
         containerDataList.append(Container(containerName, ipAddress, env, dockerComposeService))
     # Now merge the new containerData info with existing
     newContainerDataList = []
@@ -98,6 +77,44 @@ def parseComposeOutput(context):
     setattr(context, "compose_containers", newContainerDataList)
     bdd_log("")
 
+def getContainerNamesFromContext(context):
+    containerNames = []
+    for l in context.compose_error.splitlines():
+        tokens = l.split()
+
+        if len(tokens) > 1:
+            thisContainer = tokens[1]
+
+            if thisContainer not in containerNames:
+               containerNames.append(thisContainer)
+
+    return containerNames
+
+def getIpFromContainerName(containerName):
+    output, error, returncode = \
+            cli_call(["docker", "inspect", "--format",  "{{ .NetworkSettings.IPAddress }}", containerName], expect_success=True)
+    bdd_log("container {0} has address = {1}".format(containerName, output.splitlines()[0]))
+
+    return output.splitlines()[0]
+
+def getEnvironmentVariablesFromContainerName(containerName):
+    output, error, returncode = \
+            cli_call(["docker", "inspect", "--format",  "{{ .Config.Env }}", containerName], expect_success=True)
+    env = output.splitlines()[0][1:-1].split()
+    bdd_log("container {0} has env = {1}".format(containerName, env))
+
+    return env
+
+def getDockerComposeServiceForContainer(containerName):
+    # Get the Labels to access the com.docker.compose.service value
+    output, error, returncode = \
+        cli_call(["docker", "inspect", "--format",  "{{ .Config.Labels }}", containerName], expect_success=True)
+    labels = output.splitlines()[0][4:-1].split()
+    dockerComposeService = [composeService[27:] for composeService in labels if composeService.startswith("com.docker.compose.service:")][0]
+    bdd_log("dockerComposeService = {0}".format(dockerComposeService))
+
+    return dockerComposeService
+
 def allContainersAreReadyWithinTimeout(context, timeout):
     timeoutTimestamp = time.time() + timeout
     formattedTime = time.strftime("%X", time.localtime(timeoutTimestamp))

bddtests/docker-compose-1.yml bddtests/bdd-docker/docker-compose-1.yml

@@ -83,4 +83,4 @@ def bdd_log(msg):
     print("{} - {}".format(currentTime(), msg))
 
 def currentTime():
-    return time.strftime("%H:%M:%S")
+    return time.strftime("%H:%M:%S")

bddtests/docker-compose-2-tls-basic.yml bddtests/bdd-docker/docker-compose-2-tls-basic.yml

@@ -393,7 +393,7 @@ def transactionCommittedToContainersWithinTimeout(context, containers, timeout):
             .format(container.name)
 
 def responseIsOk(response):
-    isResponseOk = False;
+    isResponseOk = False
 
     status_code = response.status_code
     assert status_code == 200 or status_code == 404, \
@@ -578,12 +578,6 @@ def step_impl(context):
 
     context.peerToSecretMessage = peerToSecretMessage
 
-
-@given(u'I stop peers')
-def step_impl(context):
-    compose_op(context, "stop")
-
-
 @given(u'I start a listener')
 def step_impl(context):
     gopath = os.environ.get('GOPATH')
@@ -597,6 +591,10 @@ def step_impl(context):
 def step_impl(context):
     compose_op(context, "start")
 
+@given(u'I stop peers')
+def step_impl(context):
+    compose_op(context, "stop")
+
 @given(u'I pause peers')
 def step_impl(context):
     compose_op(context, "pause")