Merge branch 'develop' into feat/samrc

* develop:
  chore: Version bump aws-sam-translator to 1.8.0 (aws#732)
  docs: add instructions for using local version of SAM Transformer (aws#688)
  docs: Update Docker command to docker (aws#725)
  fix: Iterate over query param list
  chore: Enable Travis to run integ tests
  chore(0.6.1): SAM CLI Version bump (aws#717)
  fix(init/readme): Correct permissions for AWS CLI under requirements (aws#713)
  add pytest-mock for testing (aws#703)
  fix: allow for stdout and stderr streams to be unbufferred directly. (aws#708)
  docs: Add installation instructions for linux (Centos) (aws#670)
  fix: Updated isBase64Encoded value to bool (aws#699)
  fix: correct launch.json for nodejs debugging through VSCode (aws#704)
  docs(usage): Update how to debug Python functions using VS Code (aws#694)
  docs(Cloud9): Reset bash cache on Cloud9 (aws#693)
  docs: Updated virtualenv alias name for 3.7 in guide. (aws#706)
  chore: update aws-sam-translator to 1.7.0 (aws#682)
  feat: travis CI support for Python 3.7 (aws#679)
  docs: Update generate-sample-event-payloads link (aws#702)
  Fix: Raise error for invalid environment variables file (aws#675)

.travis.yml

@@ -1,17 +1,30 @@
 # Enable container based builds
-sudo: false
+sudo: required
 language: python
 
+services:
+  - docker
+
 python:
   - "2.7"
   - "3.6"
 
+env:
+  global:
+    - AWS_DEFAULT_REGION=us-east-1
+
+# Enable 3.7 without globally enabling sudo and dist: xenial for other build jobs
+matrix:
+  include:
+    - python: 3.7
+      dist: xenial
+      sudo: true
+
 install:
   # Install the code requirements
   - make init
 
-# Install Docs requirements
-
 script:
   # Runs unit tests
   - make pr
+  - make integ-test

DEVELOPMENT_GUIDE.rst

@@ -14,7 +14,7 @@ Environment Setup
 
 1. Install Python Versions
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
-We support both Python 2.7 and 3.6 versions.
+We support Python 2.7, 3.6 and 3.7 versions.
 Follow the idioms from this `excellent cheatsheet`_ to make sure your code is compatible with both Python versions.
 Our CI/CD pipeline is setup to run unit tests against both Python versions. So make sure you test it with both
 versions before sending a Pull Request. `pyenv`_ is a great tool to easily setup multiple Python versions.
@@ -24,16 +24,19 @@ versions before sending a Pull Request. `pyenv`_ is a great tool to easily setup
 #. Install PyEnv - ``curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash``
 #. ``pyenv install 2.7.14``
 #. ``pyenv install 3.6.4``
-#. Make both Python versions available in the project: ``pyenv local 3.6.4 2.7.14``
+#. ``pyenv install 3.7.0``
+#. Make Python versions available in the project: ``pyenv local 3.6.4 2.7.14 3.7.0``
 
 
 2. Activate Virtualenv
 ~~~~~~~~~~~~~~~~~~~~~~
 Virtualenv allows you to install required libraries outside of the Python installation. A good practice is to setup
 a different virtualenv for each project. `pyenv`_ comes with a handy plugin that can create virtualenv.
 
-#. Create new virtualenv if it does not exist: ``pyenv virtualenv 2.7.14 samcli27`` and ``pyenv virtualenv 3.6.4 samcli36``
-#. ``pyenv activate samcli27`` for Python2.7 or ``pyenv activate samcli36`` for Python3.6
+Depending on the python version, the following commands would change to be the appropriate python version.
+
+#. ``pyenv virtualenv 3.7.0 samcli37``
+#. ``pyenv activate samcli37`` for Python3.7
 
 
 3. Install dev version of SAM CLI
@@ -42,10 +45,36 @@ We will install a development version of SAM CLI from source into the virtualenv
 make changes. We will install in a command called ``samdev`` to keep it separate from a global SAM CLI installation,
 if any.
 
-#. Activate Virtualenv: ``pyenv activate samcli27`` or ``pyenv activate samcli36``
+#. Activate Virtualenv: ``pyenv activate samcli37``
 #. Install dev CLI: ``make init``
 #. Make sure installation succeeded: ``which samdev``
 
+4. (Optional) Install development version of SAM Transformer
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to run the latest version of [SAM Transformer](https://github.com/awslabs/serverless-application-model/), you can clone it locally and install it in your pyenv. This is useful if you want to validate your templates against any new, unreleased SAM features ahead of time.
+
+This step is optional and will use the specified version of aws-sam-transformer from PyPi by default.
+
+```bash
+# cd into the directory where you usually place projects and clone the latest SAM Transformer
+cd ~/projects
+git clone https://github.com/awslabs/serverless-application-model/
+
+# cd into the new directory and checkout the relevant branch
+cd serverless-application-model
+git checkout develop
+
+# Install the SAM Transformer in editable mode so that all changes you make to
+# the SAM Transformer locally are immediately picked up for SAM CLI.
+pip install -e .
+
+# Move back to your SAM CLI directory and re-run init
+# If necessary: open requirements/base.txt and replace the version number of aws-sam-translator with the
+# version number specified in your local version of serverless-application-model/samtranslator/__init__.py
+cd ../aws-sam-cli
+make init
+```
 
 Running Tests
 -------------

docs/installation.rst

@@ -18,7 +18,16 @@ environment variable to contact the docker daemon.
 -  **Windows**: `Docker
    For Windows (create an account & follow through to download from the Docker Store) <https://www.docker.com/docker-windows>`__
 -  **Linux**: Check your distro’s package manager (e.g. yum install docker)
-
+      *** for Centos 7.5 System requirements are::
+         yum install gcc zip py-pip py-setuptools ca-certificates groff  python-dev g++ make docker epel-release python-pip python-devel\
+                     python-tools
+          **run post install of above**
+          pip install --upgrade pip
+          pip install --upgrade setuptools
+          pip install --upgrade aws-sam-cli
+        *** if you want to use the lambda-local option(without running it as root) you will need to add your user to the docker group ***
+         usermod -a -G docker yourUserName
+     
 **Note for macOS and Windows users**: SAM CLI requires that the project directory
 (or any parent directory) is listed in `Docker file sharing options <https://docs.docker.com/docker-for-mac/osxfs/>`__.
 
@@ -194,6 +203,9 @@ If your AWS Cloud9 environment has a SAM CLI version < 0.3.0 installed there are
    
     # Create new symlink
     $ ln -sf $(which sam) ~/.c9/bin/sam
+    
+    # Reset the bash cache
+    $ hash -r
    
     # Verify your installation worked
     $ sam –-version

docs/usage.rst

@@ -14,7 +14,7 @@ Alternatively, you can find other sample SAM Templates by visiting `SAM <https:/
 -  `Invoke functions locally <#invoke-functions-locally>`__
 -  `Run automated tests for your Lambda functions locally <#run-automated-tests-for-your-lambda-functions-locally>`__
 -  `Generate sample event source
-   payloads <#generate-sample-event-source-payloads>`__
+   payloads <#generate-sample-event-payloads>`__
 -  `Run API Gateway locally <#run-api-gateway-locally>`__
 -  `Debugging Applications <#debugging-applications>`__
 
@@ -279,7 +279,7 @@ Both ``sam local invoke`` and ``sam local start-api`` support local
 debugging of your functions.
 
 To run SAM Local with debugging support enabled, just specify
-``--debug-port`` or ``-d`` on the command line.
+``--debug-port`` or ``-d`` on the command line. SAM CLI debug port option ``--debug-port`` or ``-d`` will map that port to the local Lambda container execution your IDE needs to connect to.
 
 .. code:: bash
 
@@ -289,6 +289,7 @@ To run SAM Local with debugging support enabled, just specify
    # Start local API Gateway in debug mode on port 5858
    $ sam local start-api -d 5858
 
+
 Note: If using ``sam local start-api``, the local API Gateway will
 expose all of your Lambda functions but, since you can specify a single
 debug port, you can only debug one function at a time. You will need to
@@ -304,11 +305,12 @@ Visual Studio Code:
    SAM Local debugging example
 
 In order to setup Visual Studio Code for debugging with AWS SAM CLI, use
-the following launch configuration:
+the following launch configuration after setting directory where the template.yaml is present
+as workspace root in Visual Studio Code:
 
 .. code:: json
 
-   {
+  {
        "version": "0.2.0",
        "configurations": [
            {
@@ -317,41 +319,124 @@ the following launch configuration:
                "request": "attach",
                "address": "localhost",
                "port": 5858,
-               "localRoot": "${workspaceRoot}",
+               // From the sam init example, it would be "${workspaceRoot}/hello_world"
+               "localRoot": "${workspaceRoot}/{directory of node app}",
                "remoteRoot": "/var/task",
-               "protocol": "legacy"
+               "protocol": "inspector",
+               "stopOnEntry": false
            }
        ]
    }
 
+Note: localRoot is set based on what the CodeUri points at template.yaml,
+if there are nested directories within the CodeUri, that needs to be
+reflected in localRoot.
+
 Note: Node.js versions --below-- 7 (e.g. Node.js 4.3 and Node.js 6.10)
 use the ``legacy`` protocol, while Node.js versions including and above
 7 (e.g. Node.js 8.10) use the ``inspector`` protocol. Be sure to specify
 the corresponding protocol in the ``protocol`` entry of your launch
-configuration.
+configuration. This was tested with VS code version 1.26, 1.27 and 1.28
+for ``legacy`` and ``inspector`` protocol.
 
 Debugging Python functions
 --------------------------
 
-Unlike Node.JS and Java, Python requires you to enable remote debugging
-in your Lambda function code. If you enable debugging with
-``--debug-port`` or ``-d`` for a function that uses one of the Python
-runtimes, SAM CLI will just map through that port from your host machine
-through to the Lambda runtime container. You will need to enable remote
-debugging in your function code. To do this, use a python package such
-as `remote-pdb <https://pypi.python.org/pypi/remote-pdb>`__. When
-configuring the host the debugger listens on in your code, make sure to
-use ``0.0.0.0`` not ``127.0.0.1`` to allow Docker to map through the
-port to your host machine.
-
-   Please note, due to a `open
-   bug <https://github.com/Microsoft/vscode-python/issues/71>`__ with
-   Visual Studio Code, you may get a
-   ``Debug adapter process has terminated unexpectedly`` error when
-   attempting to debug Python applications with this IDE. Please track
-   the `GitHub
-   issue <https://github.com/Microsoft/vscode-python/issues/71>`__ for
-   updates.
+Python debugging requires you to enable remote debugging in your Lambda function code, therefore it's a 2-step process:
+
+1. Install `ptvsd <https://pypi.org/project/ptvsd/>`__ library and enable within your code
+2. Configure your IDE to connect to the debugger you configured for your function
+
+As this may be your first time using SAM CLI, let's start with a boilerplate Python app and install both app's dependencies and ptvsd:
+
+.. code:: bash
+
+    sam init --runtime python3.6 --name python-debugging
+    cd python-debugging/
+    
+    # Install dependencies of our boilerplate app
+    pip install -r requirements.txt -t hello_world/build/
+    
+    # Install ptvsd library for step through debugging
+    pip install ptvsd -t hello_world/build/
+    
+    cp hello_world/app.py hello_world/build/
+
+Ptvsd configuration
+^^^^^^^^^^^^^^^^^^^
+
+As we installed ptvsd library in the previous step, we need to enable ptvsd within our code, therefore open up ``hello_world/build/app.py`` and add the following ptvsd specifics.
+
+.. code:: python
+
+  import ptvsd
+  
+  # Enable ptvsd on 0.0.0.0 address and on port 5890 that we'll connect later with our IDE
+  ptvsd.enable_attach(address=('0.0.0.0', 5890), redirect_output=True)
+  ptvsd.wait_for_attach()
+
+**0.0.0.0** instead of **localhost** for listening across all network interfaces and **5890** is the debugging port of your preference.
+
+Visual Studio Code
+^^^^^^^^^^^^^^^^^^
+
+Now that we have both dependencies and ptvsd enabled within our code we configure Visual Studio Code (VS Code) Debugging - Assuming you're still in the application folder and have code command in your path, let's open up VS Code:
+
+.. code:: bash
+
+    code .
+
+``NOTE``: If you don't have code in your Path, please open up a new instance of VS Code from ``python-debugging/`` folder we created earlier.
+
+In order to setup VS Code for debugging with AWS SAM CLI, use
+the following launch configuration:
+
+.. code:: json
+
+   {
+       "version": "0.2.0",
+       "configurations": [
+           {
+              "name": "SAM CLI Python Hello World",
+              "type": "python",
+              "request": "attach",
+              "port": 5890,
+              "host": "localhost",
+              "pathMappings": [
+                  {
+                      "localRoot": "${workspaceFolder}/hello_world/build",
+                      "remoteRoot": "/var/task"
+                  }
+              ]
+          }
+      ]
+    }
+
+For VS Code, the property **localRoot** under **pathMappings** key is really important and there are 2 aspects you should know as to why this is setup this way:
+
+1. **localRoot**: This path will be mounted in the Docker Container and needs to have both application and dependencies at the root level
+2. **workspaceFolder**: This path is the absolute path where VS Code instance was opened
+
+If you opened VS Code in a different location other than ``python-debugging/`` you need to replace it with the absolute path where ``python-debugging/`` is.
+
+Once complete with VS Code Debugger configuration, make sure to add a breakpoint anywhere you like in ``hello_world/build/app.py`` and then proceed as follows:
+
+1. Run SAM CLI to invoke your function
+2. Hit the URL to invoke the function and initialize ptvsd code execution
+3. Start the debugger within VS Code
+
+.. code:: bash
+
+  # Remember to hit the URL before starting the debugger in VS Code
+
+  sam local start-api -d 5890
+
+  # OR 
+
+  # Change HelloWorldFunction to reflect the logical name found in template.yaml
+
+  sam local generate-event apigateway aws-proxy | sam local invoke HelloWorldFunction -d 5890
+
 
 Debugging Golang functions
 --------------------------
@@ -561,7 +646,7 @@ Example:
    $ sam deploy --template-file ./packaged.yaml --stack-name mystack --capabilities CAPABILITY_IAM
 
 Learn More
-==========
+----------
 
 -  `Project Overview <../README.rst>`__
 -  `Installation <installation.rst>`__

requirements/base.txt

@@ -6,7 +6,7 @@ Flask~=1.0.2
 boto3~=1.5
 PyYAML~=3.12
 cookiecutter~=1.6.0
-aws-sam-translator==1.6.0
+aws-sam-translator==1.8.0
 docker>=3.3.0
 dateparser~=0.7
 python-dateutil~=2.6

samcli/__init__.py

@@ -2,4 +2,4 @@
 SAM CLI version
 """
 
-__version__ = '0.6.0'
+__version__ = '0.6.1'

samcli/commands/local/invoke/cli.py

@@ -11,6 +11,8 @@
 from samcli.commands.local.cli_common.invoke_context import InvokeContext
 from samcli.local.lambdafn.exceptions import FunctionNotFound
 from samcli.commands.validate.lib.exceptions import InvalidSamDocumentException
+from samcli.commands.local.lib.exceptions import OverridesNotWellDefinedError
+
 
 LOG = logging.getLogger(__name__)
 
@@ -94,7 +96,7 @@ def do_cli(ctx, function_identifier, template, event, no_event, env_vars, debug_
 
     except FunctionNotFound:
         raise UserException("Function {} not found in template".format(function_identifier))
-    except InvalidSamDocumentException as ex:
+    except (InvalidSamDocumentException, OverridesNotWellDefinedError) as ex:
         raise UserException(str(ex))
 
 

samcli/commands/local/lib/debug_context.py

@@ -1,6 +1,7 @@
 """
 Information and debug options for a specific runtime.
 """
+import os
 
 
 class DebugContext(object):
@@ -13,6 +14,8 @@ def __init__(self,
         self.debug_port = debug_port
         self.debugger_path = debugger_path
         self.debug_args = debug_args
+        if self.debug_port:
+            os.environ["PYTHONUNBUFFERED"] = "1"
 
     def __bool__(self):
         return bool(self.debug_port)

samcli/commands/local/lib/exceptions.py

@@ -8,3 +8,10 @@ class NoApisDefined(Exception):
     Raised when there are no APIs defined in the template
     """
     pass
+
+
+class OverridesNotWellDefinedError(Exception):
+    """
+    Raised when the overrides file is invalid
+    """
+    pass

samcli/commands/local/lib/generated_sample_events/events/apigateway/AwsProxy.json

@@ -3,7 +3,7 @@
   "resource": "{{resource}}",
   "path": "/{{path}}",
   "httpMethod": "{{method}}",
-  "isBase64Encoded": "false",
+  "isBase64Encoded": true,
   "queryStringParameters": {
     "foo": "bar"
   },