Overview

What is routo ?

Routo is a platform that provides frameworks and tools to develop and deploy cloud applications or workloads using serverless technologies. Routo works best with applications or workloads that follow 'Event Driven' or 'Asynchronous Message Processing' architectures.

Getting Started

Note : Routo currently supports Python as the programming language and AWS as the target deployment platform. Support for other languages and other Cloud providers are in works

eCommerce Order Reconciliation - A sample scenario

Acme Inc. has exposed a FTP location (an AWS S3 Bucket with appropriate permissions) into which a partner uploads orders at the end of day
Orders are in the form of CSVs each of which can contain multiple orders
Reconciliation system converts the CSV into JSON, cleansing data if needed
Order JSON is then stored in Acme Inc.’s system through a REST API call

We will now develop and deploy this workload on AWS using the command line interface of routo.

Pre-requisites

Register for a routo account here
Configure your AWS CLI with the credentials of a user with appropriate permissions. See this section for required permissions
routo CLI requires Python 3.4 or above to be installed on your machine.
We also recommend you to use a tool like virtualenvwrapper to manage VirtualEnvs and avoid polluting the main system libraries space

Installing the CLI

Start a terminal. If you have set up a virtual env, activate the env. Then type - pip install --extra-index-url=https://test.pypi.org/simple routo-cli

You will now have access to a new routo-cli command in your terminal

Type routo-cli login in the terminal and enter your routo credentials that you created earlier

$ routo-cli login $ Routo Login Email: xxxx@xxx.xxx $ Routo Password: $ Routo Login Success

You should be seeing a Routo Login Success message as shown above

Creating a acme-order-reconciliation project

Note : You should sign-in using routo-cli login before creating a project

Type routo-cli init in the terminal. Give a name and description for your project.

$ routo-cli init $ Project Name: acme-order-reconciliation $ Project Description: My first routo project $ Creating Project....... $ Done Creating Project $ Copying Templates... $ Done Copying Templates... $ Project created with folder name acme-order-reconciliation

You should be seeing a Project created with folder name acme-order-reconciliation message as shown above

Project layout

The project that was created in the previous step is a regular Python project. You should be able to open it in your favourite Python IDE. You will see the following files in the acme-order-reconciliation project -

routes.py - the main module containing the projects 'routes'. A route is a movement of messages from a source end-point to a target end-point. Along a route, messages can be enriched or transformed with the help of processors.
processors.py - this module contains custom functions that are created by you to transform or enrich messages. These functions are used in routes.py. Custom functions always take a single parameter as input - the string form of the message to be processed
.routoproject - this is a configuration file used by routo to store project meta-data
requirements.txt - if your custom functions in processors.py depend on external libraries, those dependencies should be added in this file

Code Changes

routes.py

Open routes.py in your IDE. The module comes pre-populated with a sample route and comments. Delete existing contents.
Add the following imports

from routo.dsl import Route
from processors import *

The first line imports library modules while the second imports the processor module that will contain your custom functions

Add a route that converts CSV to JSON and splits into multiple JSONs one for each order. Copy-paste the following line

r1 = Route('CSVToJSON').from_end_point('file:OrdersFTPBucket').split(csv_to_json).to('queue:OrdersJSONQueue')

The above route reads from a S3 bucket called OrdersFTPBucket , converts CSV to JSON And splits into individual order JSON by using the custom function csv_to_json and writes each of the new order json into the SQS queue called OrderJSONQueue

Add route that reads from the OrderJSONQueue and makes the API call to store orders, by using the custom store_orders function. Copy-paste the following line

r2 = Route('StoreOrders').from_end_point('queue:OrdersJSONQueue').process(store_orders)

Expose above routes using the reserved route_list variable. Copy-paste the following line

route_list = [r1 , r2]

processors.py

Open processors.py in your IDE. The module comes pre-populated with a sample custom function and comments. Delete existing contents.
Add the following external library imports

import json
import requests

Copy-paste the custom csv_to_json function

def csv_to_json(input_string):
  csv_rows = input_string.splitlines()
  final_orders = {}
  for row in csv_rows[1:]:
    columns = row.split(",")
    order_id = columns[0]
    product_name = columns[1]
    quantity = columns[2]
    order = final_orders.get(order_id, {'id':order_id, 'products':[{'name':product_name, 'quantity':quantity}]})
    product = {'name': product_name, 'quantity': int(quantity)}
    order['products'].append(product)
    final_orders[order_id] = order
  return [json.dumps(value) for value in final_orders.values()]

Copy-paste the custom store_orders function

def store_orders(order_json):
  response = requests.post('https://api.stage.routo.io/api/v1/dummy_orders',data=order_json)
  if (response.status_code != 200):
      raise Exception(response.content)

requirements.txt

Copy-paste the following line - requests==2.22.0

Deploying the project

Configure AWS credentials as described here

To deploy the project, cd into project directory and type the following -

routo-cli deploy --env you-env-name

Note: The env name that is supplied above should correspond to an env that you have used in AWS configuation step.

You will see the progress of deployment in the console. At the end of the deployment step, all AWS resources needed for this application are automatically provisioned using CloudFormation and SAM templates and the application logic is included as a part of one or more Lambda functions. In short, a fully functional application is up and running at the end of this step.

You can now sign-in to the routo Dashboard and view project status and metrics. Additionally, you can see AWS resources that have been created for your project in the Stacks menu. These labels are deep-linked to the actual AWS resource in your account.

For our sample project acme-order-reconciliation, following resources are created -

Name	Type	Function
OrdersFTPBucket	S3 bucket	FTP bucket where Order CSVs are dropped
CSVToJSONExecutor	Lambda Function	Contains the processing logic defined in `CSVToJSON` route ie., reading CSV from `OrdersFTPBucket`, splitting into multiple JSONs by using the `csv_to_json` custom function and storing each of the JSONs in `OrdersJSONQueue`
CSVToJSONDLQ	SQS Queue	Messages that cannot be processed in `CSVToJSON` route are placed in this Dead Letter Queue
OrdersJSONQueue	SQS Queue	Standard Queue containing Order JSONs
StoreOrdersExecutor	Lambda Function	Contains the processing logic defined in `StoreOrders` route
StoreOrdersDLQ	SQS Queue	Messages that cannot be processed in `StoreOrders` route are placed in this Dead Letter Queue

To test, open OrdersFTPBucket in a new tab and upload a CSV. You should be able to open CloudWatch Logs for both Lambdas and see that the Lambdas are executed without errors.

Execution Flow

The illustration below shows how the components work in the deployed application

routo components

This section describes the main components of routo

CLI

routo CLI is a Python based Command Line tool to create and deploy routo projects. Projects created using routo CLI can then be edited using your favourite IDE to add your processing logic. They are like any other source artifacts and can be version controlled in Git.

Runtime

routo Runtime is a library that executes the routes that you have defined, in the target serverless environment. The runtime is responsible for processing messages from the "From" end-point, moving it through the pipeline of "Processors", "Splitters" and "Branchers", eventually storing it in the "To" end-point. The library is packaged along with the project at the time of deployment

Dashboard

routo Dashboard is a Web interface that provides operational dashboards for your projects. The Dashboard provides useful metrics such as the current operational cost for your projects, the number of messages that have been processed etc. It also provides a useful shortcut to access all your stacks resources in once place.

Playground

routo Playground is a web application that lets you quickly test the features of routo without installing the CLI. Using Playground, you can create and edit Projects and deploy them to your Cloud Provider. Projects created in Playground are not version controlled in Git and are not available as a part of CI-CD.

routo DSL

Routo DSL (Domain Specific Language) is a simpl and expressive language to create serverless workloads. Drawing from the design patterns used in Enterprise Application Integration scenarios, it provides a rich set hgof 'building blocks' to model various message processing scenarios. primitives and transparent integration with cloud provider’s services, a Business User or a Developer can quickly create

from-end-point

The from-end-point method specifies a source end-point for your messages. It accepts an URI string in this format - type:name . Type denotes the source end-point type (queue or FTP location) while name denotes the logical name defined by you for the source.

Following table describes supported end-point types and the corresponding resources that will be created in each Cloud Provider when the route is deployed

Type	AWS Resource	Google Cloud Resource	Azure Resource
file	S3 bucket	todo	todo
queue	SQS Queue	todo	todo
http	API Gateway Endpoint	todo	todo
trigger	none	todo	todo

to

The to method specifies the target end-point for your messages. The format is identical to the from-end-point method

process

The process method

split

The split method is used to split a message into multiple messages each of which is then passed on sequentially to the next step. It accepts any user defined python function as the argument. The function must accept a string parameter and return a list of strings

# Split the order json into multiple jsons
def split_orders(in_message):
    return [json.dumps(o) for o in json.loads(in_message)['order']]

# Use the above split_order method in your route as shown below
r = Route('MySampleRoute').from_end_point('file:SourceBucket').split(split_orders).to('queue:TargetQueue')

branch

The branch method is used to conditionally route messages to target end-points. It accepts a list of 2-tuples as the argument. Each element of the list denotes a branching rule; the rule is represented as a tuple of size 2 where the first element of the tuple represents the condition and the second element represent a target end-point where the message should be sent to if the condition evaluates to True


r = Route('BranchExample')
    .from_end_point('http:/orders')
    .branch([
    (lambda m : 'CustomerX' in json.loads(m)['client-header'] , 'queue:CustomerXQueue'),
    (lambda m : 'CustomerY' in json.loads(m)['client-header'] , 'queue:CustomerYQueue'),
    ])

AWS configuration

Setting up IAM user

Sign in to AWS console
Create a custom policy AWSCloudFormationFullAccess-CustomPolicy using the following JSON

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "cloudformation:*",
            "Resource": "*"
        }
    ]
}

Create a IAM user (console access not needed) with the following permissions -

In the Security credentials tab, Create access key. Download and store the keys CSV in a secure manner

aws cli configuration

Open a terminal. Activate the virtualenv if you have created one. Type aws configure and use the access and secret key from above step.