Accelerating serverless deployment with Github Actions

News
19 Dec 2023 / by Ashith VL in Serverless

Basic setup for Serverless-AWS-Python

In this instance, Python will be employed in conjunction with AWS. Other compatible frameworks may also be utilized. Initiate the process by installing a fundamental serverless Python template 

$ serverless create --template aws-python3

This action will generate an AWS-Python template featuring a designated function named "handler.py" Subsequently, the project can be deployed on AWS by executing the following command:

Note: Prior to deploying the project, it is imperative to configure the AWS profile in Visual Studio Code. 

$ serverless deploy

The provided command will execute, deploying our function on AWS Lambda. Indeed, our foundational configuration using serverless-python has been successfully established.

Issue with serverless deployment

However, a significant challenge frequently encountered by developers pertains to the size of the deployed function on local systems. Allow us to elaborate on this issue.

Upon each deployment of Lambda functions using Serverless, the process involves zipping individual files separately and then deploying them to AWS. There's a crucial constraint: each zipped function must be under 50 MB in size.

The calculation of size is contingent on the libraries incorporated within that function. While the size typically falls below the stipulated limit, the use of specific sizable libraries such as Docker or Puppeteer can cause the package size to exceed the threshold.

To address this concern, one approach involves creating layers in Serverless. Nevertheless, deployment issues may persist over time, leading to time-consuming efforts, often without a clear indication of completion. This is precisely where GitHub Actions can prove immensely valuable.

GitHub Actions

The robust platform seamlessly integrates into your GitHub repositories, offering an intuitive CI/CD solution. Imagine a world where your workflows, from testing to deployment, are orchestrated effortlessly within your familiar GitHub environment. GitHub Actions isn't just about automating tasks; it's about streamlining your development pipeline, reducing manual interventions, and accelerating software delivery. In this section, we'll explore the simplicity and efficiency that GitHub Actions brings to the table, uncovering its potential to transform the way you build, test, and deploy code. Welcome to a world where automation meets collaboration, and your development workflows become a symphony of efficiency.

  • Let's proceed with setting up GitHub Actions step by step:
    • Begin by creating a .github folder in the root of your project. Inside the .github folder, create a workflows folder.
    • Within the workflows folder, create a file named action.yml. You can choose any preferred name for the file, but for consistency, let's use action.yml.
  • Note: File name can be anything of your preference.

Let us explain the steps involved in actions. 

  - uses: actions/checkout@v4
  - name: Set up Python ${{ matrix.python-version }}
    uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

Next up, let's jump into that branch mentioned in the actions.yml file 

  - name: Lint with flake8
  run: |
    python -m pip install --upgrade pip
    pip install flake8 pytest
  # stop the build if there are Python syntax errors or undefined names
  flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
  # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
  flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

Following that, we'll install all the Python dependencies and flake8. We'll use flake8 to check the code for any syntax errors, and if it finds any, the deployment process will come to a halt. 

  - name: Use Node.js ${{ env.NODE_VERSION }}
  uses: actions/setup-node@v4
  with:
    node-version: ${{ env.NODE_VERSION }}

After that, let's set up Node and specify its version to ensure the installation of the necessary dependencies for Node packages. 

  - name: Cache node modules
  uses: actions/cache@v3
  env:
    cache-name: cache-node-modules
  with:
    # npm cache files are stored in `~/.npm` on Linux/macOS
    path: ~/.npm
    key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
    restore-keys: |
      ${{ runner.os }}-build-${{ env.cache-name }}-
      ${{ runner.os }}-build-
      ${{ runner.os }}-

Prior to installing all the Node packages, it's advisable to check the cache. This helps avoid redundant installations and contributes to a faster deployment process. 

  - name: Install Node dependencies
  run: npm install

Now, proceed by installing all the packages listed in the `package.json` file 

  - name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v2
  with:
    aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
  aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  aws-region: us-east-2

Configure the AWS credentials, and we strongly advise against directly using credentials from a file. Instead, consider utilizing secrets or environment variables for enhanced security. 

  - name: Check if ECR repository exists
  id: check-repo
  run: |
    repository_name="serverless-ftl-ai-engine-dev"
    if aws ecr describe-repositories --repository-names $repository_name; then
      echo "::set-output name=repository_exists::true"
    else
      echo "::set-output name=repository_exists::false"
    fi

Since we're deploying functions with Docker, a new image will be generated in AWS ECR (Elastic Container Registry) with each deployment. To maintain efficiency, it's recommended to check for existing image files in ECR before proceeding. If any are found, the previous image files can be removed, as they serve no future purpose. 

  - name: Remove ECR repository
        if: steps.check-repo.outputs.repository_exists == 'true'
        run: |
          repository_name="serverless-ftl-ai-engine-dev"
          aws ecr delete-repository --repository-name $repository_name --force

In the event that an image file exists, it will remove image from AWS ECR. 

  - name: Serverless Deploy
  uses: dhollerbach/github-action-serverless-with-python-requirements@master
  env:
    AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
    AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Finally, the deployment process will commence, deploying our functions to Lambda. To verify the actions, simply push your code to GitHub, and you can monitor the entire process in the "Actions" section of your repository.

Sample code

name: project-name
on:
  push:
    branches:
      - dev
jobs:
  deploy:
    name: deploy
    runs-on: ubuntu-latest
    env:
      NODE_VERSION: 18.x
    strategy:
      matrix:
        python-version:
          - "3.11"
    steps:
      # --------- use Python to install Python dependencies and run linter, tests, etc. ---------
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Lint with flake8
        run: |
          python -m pip install --upgrade pip
          pip install flake8 pytest
          # stop the build if there are Python syntax errors or undefined names
          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics

      # --------- Use Node and NPM to install serverless-python-requirements ---------
      - name: Use Node.js ${{ env.NODE_VERSION }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: Cache node modules
        uses: actions/cache@v3
        env:
          cache-name: cache-node-modules
        with:
          # npm cache files are stored in `~/.npm` on Linux/macOS
          path: ~/.npm
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

      - name: Install Node dependencies
        run: npm install

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2
      
      - name: Check if ECR repository exists
        id: check-repo
        run: |
          repository_name="serverless-ftl-ai-engine-dev"
          if aws ecr describe-repositories --repository-names $repository_name; then
            echo "::set-output name=repository_exists::true"
          else
            echo "::set-output name=repository_exists::false"
          fi

      - name: Remove ECR repository
        if: steps.check-repo.outputs.repository_exists == 'true'
        run: |
          repository_name="serverless-ftl-ai-engine-dev"
          aws ecr delete-repository --repository-name $repository_name --force

      - name: Serverless Deploy
        uses: dhollerbach/github-action-serverless-with-python-requirements@master
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

Conclusion

Excited to share that we've successfully integrated GitHub Actions into our workflow! This significant step not only reduces deployment time but also introduces a robust Continuous Integration/Continuous Deployment (CI/CD) mechanism. This accomplishment marks a more efficient and reliable deployment process for our projects.