Easily create your own high-performance Adobe Premiere edit workstation and dedicated Adobe Media Encoder machine in the cloud (poor-man's version).

In this walkthrough we're going to bring up 2 AWS GPU instances. One will be used for your edit work and the other for rending your Premiere project on-demand.

The two instances will have a shared network drive using Amazon FSx, a fully managed native Microsoft Windows file system. This will allow you to work in Adobe Premiere and effortlessly make your assets and project available to AME.

Studio in the Cloud at re:Invent 2019

The inspiration for this comes from a reference architecture and guide that was developed by Teradici and AWS: https://aws.amazon.com/quickstart/architecture/cloud-video-editing/.

It's part of the larger idea of Studio in the Cloud and there is a great talk from re:Invent 2019 which covers all of the key concepts and services.

Below is a screenshot of the sample architecture shown at the re:Invent session linked above. It's a turn-key solution including many AWS services to enable your on-prem workforce to edit and render entirely in the cloud.

Poor-Man's Version

I've pulled apart their sample architecture to experiment with just the bare minimum required to have a dedicated workstation and render node linked with shared-storage using FSx.

Warning

Note that it is for prototyping, and for simplicity uses publicly available instances. You should consider all of AWS well-architected guidelines, for example, put the workstations in a private subnet with NAT gateway, RD gateways and lock-down security groups.

Demo Architecture

• The template above will create a VPC with 2 public subnets, 1 public route table and internet gateway.
• The 2 subnets are required by AWS Managed Microsoft Active Directory Service.
• The directory service is required by FSx to enable your workstations to access shared storage.
• FSx has its own security group to allow all traffic from your VPC and reject all other.
• The VFXHost security group, role and instance profile are used by your workstations.
• As noted in the warning above, 2 public IP addresses are assigned to your workstations.

The workstations are automatically configured to join the Microsoft Active Directory Domain and map the shared network drive provided by FSx.

Creating the Stack

Click the deep-link below go to CloudFormation with the template pre-loaded.

You must enter the following stack parameters on the next screen:

1. AdminPassword: Edit and Render workstation Administrator account password as well as the Active Directory domain password. DO NOT USE your personal password, generate a random string, this will appear in config files.
2. EBSVolumeSize: Workstation C drive size in gigabytes.
3. InstanceType: Workstation GPU instance type, currently limited to G3 class instances.
4. KeyPairName: Select the key pair from the dropdown.

When this stack is complete, you will have 2 running EC2 instance, go to the Outputs tab and you will see the public IP of the Edit and Render instance.

Now you can open the Teradici PCoIP Client, connect and install Adobe Premiere and AME on the respective instances. Unfortunately this can't be done before-hand because the installer requires sign-in.

Then start editing just like you would on your local PC, except you'll store everything in the shared folder. When you're ready, switch over to the render instance and render your project in AME just like you would locally.

The AME instance has the same shared folder mapped and it can render without taking up resources while you continue editing.

The EC2 instance comes pre-installed with GPU drivers and the Teradici Cloud Access software, refer to this https://aws.amazon.com/marketplace/pp/B07CSG43VK?ref=cns_1clkPro for more information on requirements and pricing.

Wrapping Up

Be sure to stop your running instances when you are done. These GPU instances are not cheap and you will quickly erode any savings by forgetting it for a few hours.

When you're ready to terminate the instance, just delete the stack, it will cleanup all of the resources associated with it.

If you stored any files on the FSx shared file system, be sure to create a backup and store it on S3 or your files will be lost when the stack is deleted.

About Me

I'm a software engineer and hold a Masters degree in Mechanical and Aerospace Engineering from the University of California, Irvine. My broad background helps me adapt and stay passionate about finding pragmatic solutions to complicated problems. Let's chat ackhatib@gmail.com.

AWS Elastic Beanstalk is an easy to use service that lets you deploy your applications without having to deal with infrastructure configuration. I use Elastic Beanstalk to deploy Docker containers backed by Elastic Container Service. You can choose which EB deployment method fits your needs best.

Elastic Beanstalk Deployment Methods

^{✦ Options available for both Single Instance and Load Balance/Auto-Scaled Environments
♦ Varies depending on instance batch size}

In this showcase, we are going to use the "Rolling" deployment method using 2 EC2 instances with a batch size of 1.

Note the relatively lower deploy time of "Rolling" because it uses existing instances. On the other extreme, "Blue/Green" deployments take the longest but have the least impact during a failed deployment. With exception of "All at Once", the rest provide zero-downtime deployments.

Overview of the Services

Here's the list of services you will be using, I'll go into more detail below.

• AWS Elastic Beanstalk
• Amazon RDS for Postgres
• AWS CodeCommit
• AWS CodeBuild
• AWS CodePipeline
• Amazon ECS (Container Service)
• Amazon ECR (Container Registry)
• Amazon VPC, S3 and IAM

Amazon RDS

Amazon's managed database service is fantastic and although this post is on zero-downtime deployment, database migrations in a ZDT fashion are awfully hard and the strategy depends on your requirements and constraints. I've included it in the showcase, but migrations are on you, sorry.

CodeCommit

AWS CodeCommit, as the name implies, is a code repository service and we'll be using it with Git. It will require an IAM user and SSH key pair as it is the preferred way to manage access to the repository.

CodeBuild

AWS CodeBuild is essentially a managed Jenkins service. It will build your container, run your tests and produce an artifact that will later be used during deployment.

CodePipeline

AWS CodePipeline is a continuous delivery service that will manage the release cycle, from new commits to build and through deployment in Elastic Beanstalk. This showcase just scratches the surface of what's possible with CodePipeline.

Amazon Container Service and Registry

The Elastic Beanstalk application will use ECS and ECR under the hood. There is no extra configuration needed for this. Elastic Beanstalk will pull the latest image from ECR and use ECS to deploy the containers. The EC2 instances, load balancer and DNS settings are managed entirely by Elastic Beanstalk.

Amazon S3, VPC and IAM

General familiarity with these is enough, you will just need take one extra step with IAM when setting up access to the CodeCommit repository.

Launch the Showcase

Click the deep-link below go to CloudFormation with the template ready to go.

Notes on the Setup

Refer do the diagram above:

• The public subnet is where your web servers will live.
• The private subnets are for the database. If using MultiAZ configuration, both will be used.
• The private subnets are of course, not publicly accessible. The database will only accept incoming connections from the VPC.
• The private subnets can access the internet through the Egress-only Internet Gateway.

Once your stack is created, the infrastructure is ready to receive your code. Go to the "Outputs" tab of the stack.

Open the EBApplicationEndpoint in your browser and you should see this:

Before you can push any changes to the CodeCommit repository, you have to configure the CodeCommit IAM User.

Configure CodeCommit SSH Access

You will need to generate an SSH key pair and upload the public key to the IAM user that was created for CodeCommit access.

1) Create an SSH key pair if you don't already have one

Open a terminal and run: $ ssh-keygen

2) Upload the SSH Public Key to the IAM User

Go to the IAM dashboard and open the IAM user that was created, it should be something like {YOUR_STACK_NAME}-CodeCommitUser-ABC123. You can find it in the CloudFormation stack Outputs.

Click "Upload SSH public key" and paste the contents of your public key.

3) Update Your SSH Config

Add this entry to your ~/.ssh/config file:

Host git-codecommit*
  user APK*********N64P
  IdentityFile ~/.ssh/{NAME_OF_YOUR_PRIVATE_KEY}

• Replace the user key above with the CodeCommit user's Access key ID
• Replace the identify file above with the path to your CodeCommit private key

If the host definition above is too broad, for example if you have many CodeCommit repositories, replace the wildcard * with the complete URL to the repository.

Setup Git for Existing or New Project

If starting from an existing project:

git remote add origin ssh://git-codecommit.REGION.amazonaws.com/v1/repos/YOUR_REPO_NAME

If starting a new project, clone the repo:

git clone origin ssh://git-codecommit.REGION.amazonaws.com/v1/repos/YOUR_REPO_NAME

Configure CodeBuild

You will need to create a buildspec.yml file which CodeBuild will use to build your Docker image, tag and push it into Elastic Container Registry (ECR). A sample of mine is below, I basically copied the push commands from ECR:

The sample below does the following:

1. Logs into Amazon ECR
2. Builds and tags your docker container
3. Pushes the container to ECR

version: 0.2

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --region $AWS_DEFAULT_REGION)
  build:
    commands:
      - echo Running unit tests...
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t <YOUR_CONTAINER_REGISTRY_NAME> .
      - docker tag <YOUR_CONTAINER_REGISTRY_NAME>:$IMAGE_TAG $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/<YOUR_CONTAINER_REGISTRY_NAME>:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/<YOUR_CONTAINER_REGISTRY_NAME>:$IMAGE_TAG
artifacts:
  files:
    - '**/*'
  discard-paths: yes

Replace <YOUR_CONTAINER_REGISTRY_NAME> with your ECR name.

Refer to the Build Specification Reference for CodeBuild for more information on creating your buildspec.yml with more advanced configuration.

Configure Elastic Beanstalk to use ECS

Next you need to create a Dockerrun.aws.json file which defines the container name, image and container resource specifications. Refer to the Multicontainer Docker Configuration documentation for more information.

{
  "AWSEBDockerrunVersion": "2",
  "containerDefinitions": [
    {
      "name": "<YOUR_CONTAINER_REGISTRY_NAME>:latest",
      "image": "<YOUR_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/<YOUR_CONTAINER_REGISTRY_NAME>",
      "essential": true,
      "memory": 1024,
      "cpu": 1,
      "environment" : [
      ],
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 80
        }
      ]
    }
  ]
}

Replace <YOUR_CONTAINER_REGISTRY_NAME> with the ECR name found under the CloudFormation stack Outputs. Replace <YOUR_ACCOUNT_ID with your AWS account ID. Replace <REGION> with the region in which you launched the CloudFormation stack.

Docker Container

In my example, I'm using a small nginx image just for testing.

https://github.com/nginxinc/docker-nginx/blob/master/mainline/buster/Dockerfile

Finally

After you've created the buildspec.yml, Dockerrun.aws.json (and Dockerfile if this is a new project). These files should be in the top level of your application directory.

./buildspec.yml
./Dockerrun.aws.json
./Dockerfile
./myDemoApp
 ...

Then commit and push to your CodeCommit repository and CodePipeline will start the build and deploy process.

git add . && git commit -m 'My first commit... maybe.' && git push

Rolling Deploy

Elastic Beanstalk carries out the deployment as follows (high level):

1. Takes the first instance out of service
2. Deploys new container version to the out-of-service instance
3. Ensures the health check URL returns success on the new container
4. Puts the new instance back into service if success or rolls-back if failed
5. Repeats steps 1-4 for the second instance

When the deployment is done, navigate to the EB environment URL and you should see this:

Accessing the Database

The CloudFormation template configures Elastic Beanstalk to exposes the database host, port, name, username and password as environment variables in your containers.

• RDS_HOSTNAME
• RDS_DB_NAME
• RDS_PORT
• RDS_USERNAME
• RDS_PASSWORD

Amazon Route53

Route53 isn't shown in this demo. When you're ready to point a domain to your application, create a CNAME record pointing to the application environment URL found under the CloudFormation stack Outputs.

Wrapping Up

Be mindful of the cost, you will have at least 3 instances running at all times, 2 for EB and 1 for RDS. If you delete the stack, be sure to backup the database as it will be deleted.

About Me

I'm a software engineer and hold a Masters degree in Mechanical and Aerospace Engineering from the University of California, Irvine. My broad background helps me adapt and stay passionate about finding pragmatic solutions to complicated problems. Let's chat ackhatib@gmail.com.

The goal of this project is to design a convolutional neural network to classify traffic signs. Other architectures are explored such as GoogLeNet. I started with the LeNet architecture and then added inception modules and dropout.

The goals/steps of this project are the following:

• Load the data set (Download Here)
• Explore, summarize and visualize the data samples
• Design, train and test the model architecture
• Use the model to make predictions on new images
• Analyze the softmax probabilities of the new images

Training, Validation and Testing Data Size

I used numpy to calculate summary statistics of the traffic signs data set:

• Size of training set: 34799
• Size of the validation set: 4410
• Size of test set: 12630
• Shape of a traffic sign image: 32x32 (RGB)
• The number of unique classes/labels: 43

Distribution of Training Data

Here is an exploratory visualization of the data set. The bar chart shows the distribution of training samples by class label.

Convolutional Neural Network (CNN) Model Architecture

Preprocessing Data Sets

The first step is grayscaling the images to increase performance during training.
However, there may be some cases where the color can be significant, although I
haven't explored this.

The image is also normalized to improve training performance, making the grayscale pixel numerical values between 0.1-0.9 instead of the range from 0-255.

Here is an example of a traffic sign image before and after grayscaling and normalization.

Model Architecture

My final model consisted of the following layers. I started with the LeNet architecture and then followed the paper on GoogLeNet by adding inception modules, average pooling followed by 2 fully connected layers with dropout in between both.

Training the Model

• Preprocessed the images
• One hot encoded the labels
• Trained using Adam optimizer
• Batch size: 128 images
• Learning rate: 0.001
• Variables initialize with truncated normal distributions
• Hyperparameters, mu = 0, sigma = 0.1, dropout = 0.5
• Trained for 20 epochs

Training was carried out on an AWS g2.2xlarge GPU instance. Each epoch was completed in around 24s, total training time of about 6-8 minutes.

Designing the Model

Final model results:

• Validation set accuracy: 0.97
• Test set accuracy: 0.95

The approach started with a review of LeNet and GoogLeNet architectures and those were used as a starting point to decide how to order and how many inception modules, pooling convolution and dropout layers to use.

• First started with LeNet and with minor tweaks, achieved an accuracy of about 0.89
• Reduced and switched max pooling to average pooling to get to 0.90
• Added an inception module after the first two convolutions to get to 0.90-0.93
• At this point, there were very deep inception modules (referencing GoogLeNet) and 4 fully connected layers and the model appeared to be overfitting.
• Reducing the number of fully connected layers to 2 and adding dropout between them made the largest impact and achieved around 0.97 accuracy.

Once I noticed the accuracy starting high and surpassing the minimum 0.93, I tweaked the layers by trial and error but my approach was to try and keep as much information flowing through the model while balancing the dimensional reduction. This led me to have 2 convolution layers with small patch sizes, average pooling with small patch size and do more severe reductions with the inception modules.

Test a Model on New Images

Here are some other German traffic signs that I found on the web. The signs are
generally easy to identify, I chose a few that are at angles and one that had a
"1" spray painted onto a 30km/h sign.

New Image Test Results

Here are the results of the predictions on the new traffic signs:

The model was able to correctly guess 8 of the 10 traffic signs, which gives an accuracy of 80%. I was impressed that the spray-painted sign still classified correctly even though the clear 100km/h sign was wrong.

The bicycles crossing and bumpy road signs do look very similar at low resolution, so I can understand that wrong prediction.

Model Certainty

The following shows the top 5 softmax probabilities for each sign.

Visualizing the Model Layers

It's interesting to see the layers of the network as it progresses. The convolution layers look the most interesting, while the average pooling layers start to look overly simplified. However, the average pooling images only show one layer in a very deep set.

Conclusions and Discussion

Having LeNet and GoogLeNet as starting points was very helpful. There are still many areas for improvement by tweaking variables, moving/adding/removing layers and augmenting the data set. I noticed that when applying the GoogLeNet architecture directly to this data set, the training time was very high and accuracy plateaued quickly which could indicate over/underfitting. But in this case, I believe there were too many parameters, I found better performance by reducing the depth of the inception layers and adding dropout.

Check out the GitHub repo for the code and Jupyter Notebook.

This project focussed on detecting vehicles and tracking them in a video.

Training images of cars and non-cars features are extracted using Histogram of Oriented Gradients (HOG). I then train a model using Support Vector Machine (SVM) to classify images as car or non-car. Finally, a heat map of detections is used to draw the final bounding box on the image.

Vehicle Detection and Tracking

This project focussed on detecting vehicles and tracking them in a video. Training images of cars and non-cars are run through a Histogram of Oriented Gradients (HOG) feature extraction pipeline. I then train a model using Support Vector Machine (SVM) to classify images as car or non-car.

The sliding window technique is used to classify multiple sub-sections of a frame, then all of the potential detections are converted to a heat map. Finally, a bounding box of the car is drawn on the frame.

This is an overview of the vehicle tracking project that also included advanced lane finding. The combined result is very interesting to see.

Goal of the Project

• Perform a Histogram of Oriented Gradients (HOG) feature extraction on a labeled training set of images and train a Linear Support Vector Machine (SVM) classifier
• Apply a color transform and append binned color features, as well as histograms of color, to the HOG feature vector.
• Normalize your features and randomize a selection for training and testing.
• Implement a sliding-window technique and use the trained classifier to search for vehicles in images.
• Run pipeline on a video stream (testing with test_video.mp4 and later implement on full project_video.mp4) and create a heat map of recurring detections frame by frame
• Use the heatmap to reject outliers and follow detected vehicles.
• Estimate a bounding box for vehicles detected.

I started by reading in all the vehicle and non-vehicle images.  Here is an example of one of each of the vehicle and non-vehicle classes:

I then explored different color spaces and different skimage.hog() parameters:  orientations, pixels_per_cell, and cells_per_block.

I selected some random images from each of the two classes and displayed them to get a feel for what the skimage.hog() output looks like.

Histogram of Oriented Gradients (HOG)

The code for this step is contained in the second code cell of the IPython notebook under Utilities.

Here is an example using the YCrCb color space and HOG parameters of orientations=8, pixels_per_cell=(8, 8) and cells_per_block=(2, 2).

Choosing HOG Parameters

I tried many combinations of parameters and trained the classifier with each set and achieved 95%-98% accuracy during training. I found that using spatial and color histograms weren't adding any value to the classifier and were slow. I also found that YUV color space provided the best results (and least false positives during testing). I continued to tune just the HOG parameters and achieved good training results with the following:

Training a Linear Support Vector Machine (SVM)

I trained a linear SVM purely on HOG features. After much testing, I didn't see any added value in spatial or color histograms for this project and they added a lot of overhead. However, they may be useful in different lighting conditions or road conditions.

Sliding Window Search

Searching the images was carried out by first cropping the search area to just the road. Then a HOG feature extraction is taken on the region once per frame. Within the region, HOG features are extracted from each window and run through the SVM predictor.

In order to improve the precision of the bounding box, I divided up the region from top to bottom, smaller windows were confined to the top of the region where cars would appear the smallest. The window size then increased moving towards the foreground where cars appear larger.

Ultimately I searched on 6 subregions and scales using YUV 3-channel HOG features. To reduce the jitter in the bounding box, I also keep track of the last set of detected bounding boxes. I use these to filter the next frame and increase the heat map region. Finally, I optimized the search to achieve nearly 2 frames per second during processing.

Video Implementation

Here is the final video implementation including lane line detection. The pipeline applies the advanced lane finding techniques and vehicle detection and tracking to each frame of the video.

Final Project Video with Lane Detection

Detection and False Positives

I recorded the positions of positive detections in each frame of the video. I also track and add the previously detected positions. From the positive detections, I created a heat map and then thresholded that map to identify vehicle positions. I then used scipy.ndimage.measurements.label() to identify individual blobs in the heat map. I constructed bounding boxes to cover the area of each blob detected.

Conclusions and Discussion

I tried many combinations of color spaces for HOG feature extraction and color histogram. I found that HOG worked well on the lightness L or Y channels. However, without spatial or color features, HOG on 1 channel resulted in many false positives. I tried different orients and pixels per cell for HOG extraction to find values that produced good training results but were general enough to avoid over-fitting.

Spatial and color histogram features weren't the focus of my adjustments, I found that they weren't adding any significant value to the classifier and just increased processing time. There are still some false positives that make it through and the bounding box is constantly adjusting. I did my best to reduce the jitter by banding the search region for each search window size and using previously detected regions to augment the heat map.

There is still much improvement that can be done to reduce the jittering of the bounding box. Overall, I think the classifier performed well and the search could be further optimized by augmenting the training dataset.

Check out the Project Github Repo to see the full implementation and code.

Programming a Real Self-Driving Car.

This is the project repo for the final project of the Udacity Self-Driving Car Nanodegree.

Team: Experienced Drivers

Implementation

The starter repo provided the skeleton of the architecture in Figure 1 with all of the ROS nodes stubbed out. There are three ROS nodes in this diagram that we implemented.

• Waypoint Updater Node
• DBW Node
• Traffic Light Detection Node

Waypoint updater

The waypoint updater node publishes waypoints starting from the car's current position up to some threshold in front of the car. These waypoints are published to the final_waypoints topic.

The waypoint updater node subscribes to the following topics:

• /current_pose emits current position of the car.
• /base_waypoints emits planned route of the car as an array of waypoints.
• /current_velocity emits current velocity of the car.
• /traffic_waypoints emits information about the state of the traffic light ahead

Pose Callback

This is the callback function that receives the current position of the car.

Whenever it receives a new position of the ego-car, it searches for the waypoints closest to the car.

If the car's position has changed since the last update, the node shifts the window of active waypoints ahead of the car and remove those behind the car.

If the car's position hasn't changed "much" between updates, we don't publish updated waypoints to avoid sending duplicate data.

This has an exception where if the car is trying to resuming from a stopped state, the node publishes updated waypoints even when our position hasn't changed "much" this is to prevent the car from stalling.

Traffic Light Callback

This is the callback function that receives the closest red light waypints.
If the value is -1 meaning no red light nearby, we will publish the received /base_waypoints. If there is a visible red light ahead, the car needs stop at the defined stop line and resuming when the light turns green.

We achieve both stopping and resuming by setting the velocity property of the waypoints in front of the car.

For stopping we linearly decrease the speed of each waypoint using the car's current position and the light's top line to determine the rate of decrease. To make sure it does not overshoot we set extra_brake_wps so that if the car overshoots by accident it will still try to stop and not continue through the intersection.

For resuming we gradually bring the car back to the pre-configured target velocity set by the launch file. We again linearly increase the waypoint velocity value from 0 to target velocity.

Some flags like self.stopping and self.resuming are present to track the current state of the car.

Calculating distance between waypoints

To decelerate/accelerate the car, we need to calculate a linear relationship between the start/end waypoints and speed. First we calculate distance between waypoints:

def distance(waypoints, wp1, wp2):
    dist = 0
    dl = lambda a, b: math.sqrt((a.x-b.x)**2 + (a.y-b.y)**2 + (a.z-b.z)**2)
    for i in range(wp1, wp2+1):
        dist += dl(waypoints[wp1].pose.pose.position, waypoints[i].pose.pose.position)
        wp1 = i
    return dist

Then we can use this to calculate the speed of each waypoint leading up to a stop or target velocity.

def decelerate_waypoints(waypoints, closest_idx):
    newWps = []
    for i,wp in enumerate(waypoints):
        p = Waypoint()
        p.pose = wp.pose

        stop_idx = max(stopline_wp_idx - closest_idx - 2, 0)
        dist = distance(waypoints,i,stop_idx)
        vel = math.sqrt(2 * MAX_DECEL * dist)
        if vel < 1.0:
            vel = 0.0

        p.twist.twist.linear.x = min(vel, wp.twist.twist.linear.x)
        newWps.append(p)
    return newWps

DBW (Drive-By-Wire)

We update dbw_node.py and twist_controller.py. Drive-by-wire (DBW) node takes as input the target twist and publishes the driving commands: throttle, brake and steer.

The DBW node subscribes to the following topics:

• /vehicle/dbw_enabled whether DBW is enabled or disabled during manual override
• /current_velocity current velocity
• /twist_cmd target twist which includes linear and angular velocity

The DBW node publishes to:

• /vehicle/steering_cmd
• /vehicle/throttle_cmd
• /vehicle/brake_cmd

In twist_controller.py, we create 3 controllers:

• accel_controller is a PID controller to control the target acceleration.
• lowpass_filter is a Low Pass Filter to smooth out and reduce the noise of the target acceleration calculated by the accel_controller.
• yaw_controller is a Yaw Controller to calculate the target steering based on target and current linear and angular velocity.
• throttle_controller is another PID controller that takes acceleration output from the lowpass_filter and calculates the actual throttle command.

We reset the throttle_controller when manual override is active to prevent errors from building up in the controller.

If acceleration is negative, we calculate brake based on vehicle status specified by the input controller.

Proportional-Integral-Derivative (PID) controller doesn't get any simpler than this:

class PIDController:

    def __init__(self, kp, ki, kd, mn=float('-inf'), mx=float('inf')):
        self.kp = kp
        self.ki = ki
        self.kd = kd
        self.min = mn
        self.max = mx
        self.int_val = 0.0
        self.last_error = 0.0

    def step(self, error, sample_time):
        i = self.int_val + error * sample_time;
        d = (error - self.last_error) / sample_time;
        self.last_error = error

        val = self.kp * error + self.ki * i + self.kd * d;

        if val > self.max:
            val = self.max
        elif val < self.min:
            val = self.min
        else:
            self.int_val = integral

        return val

Traffic Light Detection

We updated tl_detector.py and added cv_method.py

Resubmission notes:
cv_method.py did not work for Carla driving. Reason being the filtering if too dependent on the lighting condition and would change a lot due to weather.
We have not switched back to our original NN method. This will be described in later section and code in carla.py

Traffic Light Detector

The traffic light detection node is responsible for recognizing traffic lights and determining their state eg: red, yellow or green.

The traffic light detection node subscribes to the following topics:

• /current pose current position of the car
• /base_waypoints planned route
• /vehicle/traffic_lights positions of all traffic lights
• /image_color color of the current light in front of the car

Basically, it detects if there is a visible red light ahead or not.  It will publish its result to /traffic_waypoint.

Much of the work was dedicated to the process_traffic_lights function.
Here we first computed the position of all the lights and stopping line before it.
Then for each updated current position, we check if there is an upcoming traffic light within 60 waypoints distance. If so we ask the classifier what color. If it is RED, we pass it through STATE_COUNT_THRESHOLD and publish it to the waypoint updater node through /traffic_waypoint topic.

The traffic light classifier takes an image as input and returns if there is a red light present. We have been trying out both Neural Network based or classical vision based algorithms. We have decided to use the classical vision method. The reason is due to the fact that we have been using the VM provided by the course and it does not have enough computation resources to run NN in tensor flow on real-time.
We have working NN that is able to detect the light correctly, but by the time it is done, the car has already driven past the traffic light.

This leads to our classical vision method using OpenCV functions.
It is implemented in cv_method.py

Pure OpenCV Approach

The idea behind our algorithm is to detect a red filled circle within an image.
We use cv2.HoughCircles to achieve it, however, there are more things to consider here. The source of this method is from this blog post Reference.

It is able to detect a red colored circle. The algorithm described in the blog post is able to detect a red circle in the simulator. However, it is not the case for ROSBAG real-life image.

Color is different in real life due to lighting.

The color threshold set for the simulator is almost perfect red. While in ROSBAG it is very bright and closer to orange.

Solution: we have added separate cv2.inRange function to cater both the real life and simulator.

HoughCircles does not care if a circle is filled or not.

HoughCirlces are circles detection but we need the circle to be filled for red light detection. This leads to a lot of false positive as well.

Solution: After HoughCircles returns a list of circles, I applied a cv2.countNonZero to see how filled it is, if it is below a certain value, it is not counted as a detected light. Notice in the image below, only the green circle is the circle that is filled and counted as a red light.

False Positives

The environment, due to the sunlight, has a lot of bright red spots that have exactly the same color as the traffic light. This results in a lot of false positives.

Solution: STATE_CHANGE_THRESHOLD is increased to 5 so that we can allow more false positives. Also, we increase minimum distance between two circles in HoughCircles function so that false positives that clustered together can be eliminated. Notice below some false positive even after the filled ratio threshold will still be unavoidable.

Stopping Too Late

Another problem we have seen is in the simulator, it the traffic light changes to red right before we pass the stop line, we might not be able to stop the car on time.

Solution: Deliberately increase the threshold to detect yellow color as well. So the car can be made aware it is going to be RED soon and slow down in advance.
As a result: in our project loginfo, we will output only two scenarios when we publish upcoming red light after state count thresholding. They are:

[RED or YELLOW] vs [GREEN or UNKNOWN] as shown below:

[INFO] [1522422382.484703]: GREEN or UNKNOWN 
[INFO] [1522422383.484754]: GREEN or UNKNOWN 
[INFO] [1522422383.484896]: GREEN or UNKNOWN 
[INFO] [1522422385.484902]: RED or YELLOW 
[INFO] [1522422385.484941]: Breaking distance to light: 53.1394
[INFO] [1522422385.484959]: RED or YELLOW
[INFO] [1522422385.484967]: RED or YELLOW
[INFO] [1522422386.485123]: RED or YELLOW

Carla

Due to hardware constraints on our end, we did not use a neural network classifier initially. But the pure CV approach was proving insufficient when running on the actual car. We decided to switch back to using a neural network classifier and optimize our usage.

We used a pre-trained model from tensorflow detection model zoo.
We picked ssd_mobilenet_v1_coco since it is the fastest.

This model is trained w.r.t. COCO dataset where there traffic_light is a class. So by passing an image into the network, boxes are returned to indicate object detected and type of the object.
It is a single-shot detector (SSD) using mobilenet architecture.

Inputs are 300x300 for the network.
So we provided 6 regions of interest with size 300x300, assuming our images are 600x800.

We loop over the 6 regions of interest and feed it into the classifier.
If a traffic light is detected, we will stop searching to save time.

Once a traffic light has been boxed. We will convert the cropped image to HSV like the pure CV approach used previously.

Looking at the V color value, we filter the very bright pixels and get an average height of it.

We also filter out the very dark pixels and get an average height of it.

If the light is on top and darkness is at the bottom, then it is a red light.

brightness = cv2.cvtColor(traffic_light, cv2.COLOR_BGR2HSV)[:,:,-1]
light_h, _ = np.where(brightness >= (brightness.max() - 5))
dark_h, _ = np.where(brightness <= (brightness.max() - 50))
total_h = traffic_light.shape[0]
combined_h = (light_h.mean() + (total_h - dark_h.mean())) / 2
light_ratio = combined_h / total_h

if light_ratio < 0.53: # A larger value includes more YELLOW
    return TrafficLight.RED
if light_ratio > 0.60:
    return TrafficLight.GREEN
return TrafficLight.YELLOW

This method assumes the traffic light setup is vertical and RED always on top.

We try to treat YELLOW lights as RED to be cautious and attempt to slow down at yellow lights.

Summary

With the above implementation, we successfully ran our algorithm in both simulator and ROSBAG files. Running on the actual car was challenging and there were some cases when red or green lights were missed, but overall, the project was fun, challenging and we all learned a lot about each component as well as putting everything together in a functioning system.

Check out the GitHub repo for instructions on running the simulation.

Artificial Neural Networks and Machine Learning techniques have been used in the past to predict NFL game outcomes.

Some of the research is from years ago at a time when AI and ML models were arguably more difficult to implement. Now we have access to larger data sets and relatively cheap cloud computing resources. The NFL keeps extensive statistics on the player and team performance by game, play by play and player by play.

I chose to focus on NFL games because of the massive popularity of the sport in the United States, the relative ease of gathering data and the extensive amount of data gathered by organizations.

Introduction

Artificial Neural Networks and Machine Learning techniques have been used in the past to predict NFL game outcomes, see [1, 2, 3], and many more with a simple Google search. Some of these papers were written years ago at a time when AI and ML models were arguably more difficult to implement. Now we have access to larger data sets and relatively cheap cloud computing resources. The NFL keeps extensive statistics on the player and team performance by game, play and play and player by play [4]. Frameworks such as TensorFlow, PyTorch, Keras and others let us quickly get started, experiment and explore data and models without getting bogged down in implementation details (that comes later).

Gathering the Data

Box score data is available from a few places, I evaluated the following:

• NFL
• ESPN
• FantasyData

I used a script to pull the 2017 regular season games from the NFL website which includes the team and player stats. You can download a sample game record to get an idea of what and how much data is available. I've included a snippet here:

{
  "away": {
    "abbr": "KC", // team
    "score": {
      "1": 7, // quarter and score
      ...
    },
    "stats": {
       "defense": {
        "00-0123456": {
          "ast": 4, // assists
          "ffum": 0, // fumbles
          "int": 0, // interceptions
          "name": "J.Smith", // player name
          "sk": 0, // sacks
          "tkl": 4 // tackles
        }
      },
      "fumbles": {...},
      "kicking": {...},
      "passing": {...},
      "receiving": {...},
      "rushing": {...},
      "team": {
        "pen": 15, // number of penalties
        "penyds": 139, // penalty yards
        "pt": 6, // number of punts
        "ptavg": 42, // net punting avg
        "ptyds": 262, // punting yards
        "pyds": 352, // net passing yards
        "ryds": 185, // net rushing yards
        "top": "30:14", // time of possession
        "totfd": 26, // total first downs
        "totyds": 537, // total net yards
        "trnovr": 1 // turnovers
      }
    }
  },
  "drives": {
    "1": {
      "end": {
        "qtr": 1,
        "team": "NE",
        "time": "12:08",
        "yrdln": "KC 2"
      },
      "fds": 5, // first downs
      "numplays": 13,
      "penyds": 12, // penalty yards
      "plays": {
        "118": {
          "desc": "J.White to NE 43 for 8 yards ...",
          "down": 3,
          "players": {
            "00-0023456": [
              {
                "clubcode": "KC",
                "playerName": "J.Smith",
                "sequence": 5,
                "statId": 82,
                "yards": 0
              }
            ]
          },
          "posteam": "NE",
          "qtr": 1,
          "sp": 0,
          "time": "14:14",
          "ydsnet": 73,
          "ydstogo": 2,
          "yrdln": "NE 35"
        }
      ...
    }
}

My dataset contains 256 games from the 2017 regular season. Each game record has hundreds of data points that are potentially useful in predicting the outcome of a game. To get a better understanding of the data, I'll plot some of the features against the target variable, home win.

Exploring the Data

First, I'm going to use features that have been shown to be significant in [1, 2, 3] for predicting game outcome.

• Total yardage differential
• Rushing yardage differential
• Passing yardage differential
• Time of possession differential (in seconds)
• Turnover differential
• Home or away

Total Yardage Differential and Time of Possession

The first plot in Figure 1 shows the relationship between home win outcome and total yardage differential and time of possession differential. Observe the trend from the bottom right corner to the top right, as the time of possession and total yardage differential increase, there is a clear trend toward a winning outcome.

import matplotlib.pyplot as plt

ax = plt.figure().gca()

for i in range(X.shape[0]):
    marker = '^' if y[i] == 1 else 'v'
    color = 'g' if y[i] == 1 else 'r'
    ax.scatter(X[i,0], X[i,2], s=50, color=color, marker=marker)

ax.set_title("Home Team Wins (Green) vs Losses (Red)")
ax.set_xlabel("Total Yards Differential")
ax.set_ylabel("Time of Possession Differential")

plt.show()

Total Yardage, Time of Possession and Turn Overs

In Figure 2, I add a 3rd feature: turn over differential. Plotting these 3 features were done in [2], observe the separation of the two win outcomes.

import matplotlib.pyplot as plt

ax = plt.figure().gca(projection='3d')

for i in range(X.shape[0]):
    marker = '^' if y[i] == 1 else 'v'
    color = 'g' if y[i] == 1 else 'r'
    ax.scatter(X[i,2], X[i,0], X[i,3], s=50, color=color, marker=marker)

ax.set_title("Home Team Wins (Green) vs Losses (Red)")
ax.set_xlabel("Time of Possession Differential")
ax.set_ylabel("Total Yards Differential")
ax.set_zlabel("Turnover Differential")

plt.show()

PCA for Visualization

The final plot is of the first 2 principal components. Principal Component Analysis (PCA) can be used as part of preprocessing, it reduces the dimensions of the feature space in a way that maximizes variance.

PCA also very useful to visualize data, in this case, I take a 9-dimensional feature space and reduce it to 2 dimensions. This lets us plot Figure 3, observe a similar separation of data points as the above.

Which Features are Important?

When I initially looked at the data, I didn't have any prior knowledge of how to rank features as I have a very shallow knowledge of the game. During my research, I found a few techniques, in [2] PCA is used to weight the most important features in order to optimize training. I've seen the technique of using an ANN with L1 regularization on the weights to eliminate features that aren't important to the learning task, eg: setting weights exactly to zero.

In [1] I found an interesting visualization of a correlation heat map. It's a quick way to see how features correlate with each other. Plotting a correlation heat map is relatively easy. In Figure 4, I've generated one using Pearson's method. This is a measure of how well each feature pair linearly relates.

For the purpose of a richer correlation heat map, I've included more features in Figure 4:

• Total Yards Differentials
• Rushing Yards Differential
• Possession Time Differential
• Turnover Differential
• Passing Yards Differential
• Penalty Yards Differential
• Penalties Differential
• Total First Downs Differential

import pandas as pd
import seaborn as sns

df = pd.DataFrame(data, columns=[
    'Total Yards Differentials',
    'Rushing Yards Differential',
    'Possession Time Differential',
    'Turnover Differential',
    'Passing Yards Differential',
    'Penalty Yards Differential',
    'Penalties Differential',
    'Total First Downs Differential',
    'Home Win'
])

corr = df.corr(method='pearson', min_periods=1)  # or 'spearmans'

sns.heatmap(corr, xticklabels=corr.columns, yticklabels=corr.columns)

Observe in Figure 4 the following positive correlations to Home Win:

• Possession Time Differential
• Yardage Differential
• Total First Down Differential

This seems reasonable, as these features increase (w.r.t. home team), the home team is more likely to win. Note that the correlation coefficients driving the heat map colors are based on a linear fit of the features. There isn't a strong correlation but there is some, I'll take this into account when testing models later.

An obvious negative correlation is turnover differential. But interestingly no apparent correlation with penalties, perhaps because both teams receive penalties throughout the game.

Conclusion

This was a first look at the data available. I didn't explore all the features as I want to first baseline the research I've done so far. In subsequent posts, I'll explore more features that were included in [4, 5] such as travel distance, the relative strength of each team, weather and previous encounter outcomes.

More Frameworks and Tools

The data gathering and exploration were carried out in a Jupyter notebook running in FloydHub. It's a great platform that provides a managed environment (not limited to Jupyter notebooks). I was able to quickly switch my environment between a CPU and GPU backed instance. They offer data set management and versioning that is easily mounted into your session. I personally use it for all of my prototypes and quick data analysis. It frees you from all of the setup and configuration. The environment comes primed with all of the major machine learning frameworks and data tools, in less than a minute it's ready to go.

• FloydHub (Easy and accessible platform for building ML/AI models)
• NumPy (General purpose data manipulation)
• scikit-learn (Data analytics and machine learning library)
• pandas (Used to create the correlation coefficient matrix)
• seaborn (Used to generate the correlation heat map)

References

1. Pablo Bosch, "Predicting the winner of NFL-games using Machine and Deep Learning.", Vrije universiteit, Amsterdam, Nederland Research Paper Business Analytics , 2018
2. Joshua Kahn, "Neural Network Prediction of NFL Football Games", 2003
3. Jim Warner, "Predicting Margin of Victory in NFL Games: Machine Learning vs. the Las Vegas Line", 2010
4. Rory P. Bunker and Fadi Thabtah, "A machine learning framework for sport result prediction", Auckland University of Technology, Auckland, New Zealand, 2017
5. Jack Blundell, "Numerical algorithms for predicting sports results", 2009

In this showcase were going to use the following:

• Recently released, still in beta, HTTP API (ApiGatewayV2)
• Auth0 to handle authentication for the routes
• CloudFormation to create the resources and provide a template to build from

Setup Auth0

Setup your authentication service, I'm using Auth0, use any provider you like as long as they support JWT.

After creating an account at Auth0, navigate to APIs on the left. Then click "Create API"

Fill out the details in the popup that appears. After saving you should see the following. Take note of the "Identifier", that is your audience that will be used later.

Then view the details of the application that was created automatically for your API.

Take note of the "Domain", this is your Issuer URI that will be used later. For Auth0, this is https://<TENANT_ID>.auth0.com/.

The CloudFormation Stack

Now that your authentication provider is ready and you've noted the audience and issuer URI, you can launch the CloudFormation stack.

You will be asked to provide the Audience and Issuer URI that you should have from setting up the authentication provider above.

CloudFormation will create the following resources:

• API Gateway V2 HTTP (beta)
• $default deployment stage
• Custom authorizer using your issuer and audience from Auth0
• Testing route /test
• Testing route integration using AWS Lambda
• Testing AWS Lambda function
• Testing function IAM Role to allow logging to CloudWatch
• IAM Role to allow API Gateway to invoke the integration Lambda functions

There are 2 important items to note that cost me a lot of time debugging.

Pitfall #1: Ensure API Gateway has Proper Permissions

You must use an Invoke IAM Role on your route integration or add the appropriate Lambda resource permissions to allow API Gateway to invoke your Lambda function. Incorrect permissions will cause API Gateway to return 500 errors.

The template uses this method, I prefer the invoke IAM Role because it's easier to manage.

If not using an Invoke IAM Role, you must ensure the Lambda resource permissions allow your API Gateway to invoke the function. If you're creating the integration in the console, this is done automatically. Note, the template does not use this method.

Pitfall #2: Ensure Proper Lambda Handler Response

If your API Gateway and Lambda permissions are correct but you're still getting 500s, ensure the Lambda function response is correct.

When using the nodejs Lambda runtime, take care to use either async or callback in the handler, see https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html.

exports.handler = async function(event, context) {
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!')
  };
  return response;
}

Also ensure the response is JSON with a statusCode and string body (if applicable), see https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html#api-gateway-proxy-integration-create-lambda-backend.

Testing the API

Once the stack is complete, you can send a request to the test route /test. To get an access token, navigate to your Auth0 API and go to "Test". See the screenshot below, run the command and copy the access token.

With the access token, send a GET request to the test route and include the Authorization header but DO NOT include "Bearer" before the token.

Pitfall #3: Don't Include Bearer in the Authorization Header

When adding the authorization header, DO NOT include "Bearer" before the token string, see https://forums.aws.amazon.com/thread.jspa?threadID=240420. Your authorization header should look like this:

{
  "Authorization": "eja123..."
}

You will be sending a request to something like this URL:

https://tju42ljd6a.execute-api.us-west-2.amazonaws.com/test

Wrapping Up

The CloudFormation stack you created is a good starting point to create more routes with Lambda integrations. Any changes you make should be done through the CloudFormation template to avoid drift and manual changes.

There is no passive cost for creating and running the stack. You pay for usage of API Gateway and Lambda.

About Me

I'm a software engineer and hold a Masters degree in Mechanical and Aerospace Engineering from the University of California, Irvine. My broad background helps me adapt and stay passionate about finding pragmatic solutions to complicated problems. Let's chat ackhatib@gmail.com.

Build your own high-performance remote workstation in AWS where you can easily work with high-res content in Premiere.

You won't need to buy any hardware, you just need a decent internet connection and a cheap laptop.

The workstation will use an AWS Marketplace AMI by Teradici. It runs Windows Server 2016 with the Teradici Cloud Access software pre-installed. Both the instance and Teradici host software are per-hour pricing.

Some of the benefits of this setup:

• Work from anywhere, using a cheap laptop using a thin client and still do heavy-lift editing on a powerful PC
• Save on equipment costs, and instead, rent state-of-the-art and pay only for what you use
• Get cheap and virtually unlimited storage for your high-res content using S3

Preparation

You should do the following before going further (if you haven't already):

1. Create an AWS account: https://aws.amazon.com/
2. Subscribe to the Windows Server GPU AMI: https://aws.amazon.com/marketplace/pp/B073WHLGMC
3. Subscribe to the Teradici AMI: https://aws.amazon.com/marketplace/pp/B07CSG43VK?ref=cns_1clkPro
4. Download Teradici's Software Client: https://docs.teradici.com/find/product/software-and-mobile-clients
5. Signup for an Adobe Creative Cloud Account

Studio in the Cloud Overview

This walkthrough is based on a reference architecture and guide that was developed by Teradici and AWS: https://aws.amazon.com/quickstart/architecture/cloud-video-editing/.

It's part of the larger idea of Studio in the Cloud and there is a great talk from re:Invent 2019, the link starts at virtual workstations.

Above is the reference architecture shown in the video. As you can see it's a turn key solution including workstations, active directory, remote desktop gateways, NAT gateways and other supporting services.

Poor-Man's Version

In this walkthrough, we're going to focus on just the GPU workstation and required infrastructure to support it.

Warning

Note that it is for prototyping, and for simplicity uses a publicly available instance. You should consider all of AWS well-architected guidelines, for example, put the workstation in a private subnet with NAT gateway, RD gateways and lock-down security groups.

The architecture above has a VPC with one public subnet, route table and internet gateway. Your workstation will be placed in the public subnet and assigned an Elastic IP address.

The workstation IAM role and instance profile will give your host access to S3 resources later. The security group has open ports 3389 (RDP) and 4172 (Teradici).

Creating the Stack

Click the deep-link below go to CloudFormation with the template pre-loaded.

You must enter the following stack parameters on the next screen:

1. AdminPassword: Workstation Administrator account password.
2. EBSVolumeSize: Workstation C drive size in gigabytes.
3. InstanceType: Workstation GPU instance type, currently limited to G3 class instances.
4. KeyPairName: Select the key pair from the dropdown.

When this stack is complete, you will have a running EC2 instance, go to the Outputs tab and you will see the public IP that was assigned:

The EC2 instance comes pre-installed with GPU drivers and the Teradici Cloud Access software, refer to this https://aws.amazon.com/marketplace/pp/B07CSG43VK?ref=cns_1clkPro for more information on requirements and pricing.

Now you can open the Teradici PCoIP Client and connect to your instance.

Enter "Administrator" as the username and the password you provided when creating the stack.

After you log in, you will need to install Adobe Creative Cloud Desktop Application. Unfortunately this can't be done before-hand because the installer requires sign-in.

Once you've signed in and installed Adobe Premiere, you can start editing just like you would on your local PC. Of course, getting files to and from the instance would require some care. In the next post, I'll go over how to setup shared storage.

Wrapping Up

Be sure to stop your running instance when you are done. These GPU instances are not cheap and you will quickly erode any savings by forgetting it for a few hours.

When you're ready to terminate the instance, just delete the stack, it will cleanup all of the resources associated with it.

If you stored any files on the instance, be sure to offload them to S3 or your local machine as they will be lost when the stack is deleted.

About Me

I'm a software engineer and hold a Masters degree in Mechanical and Aerospace Engineering from the University of California, Irvine. My broad background helps me adapt and stay passionate about finding pragmatic solutions to complicated problems. Let's chat ackhatib@gmail.com.

When creating AWS CloudFormation templates that have DB instance type parameters, populating the allowed values list to validate the instance type string can be tedious.

Here are some sets of DB instance types that can be copy-pasted into the "AllowedValues" field of your AWS CloudFormation DB instance type parameter.

View more info on these instance types: Amazon RDS Instance Types

My Most Commonly Used DB Instance Set

[
  "db.t3.micro", "db.t3.small", "db.t3.medium", "db.t3.large", "db.t3.xlarge", "db.t3.2xlarge",
  "db.m5.large", "db.m5.xlarge", "db.m5.2xlarge", "db.m5.4xlarge", "db.m5.12xlarge", "db.m5.24xlarge"
]

General Purpose

[
  "db.t3.micro", "db.t3.small", "db.t3.medium", "db.t3.large", "db.t3.xlarge", "db.t3.2xlarge",
  "db.t2.micro", "db.t2.small", "db.t2.medium", "db.t2.large", "db.t2.xlarge", "db.t2.2xlarge",
  "db.m5.large", "db.m5.xlarge", "db.m5.2xlarge", "db.m5.4xlarge", "db.m5.12xlarge", "db.m5.24xlarge",
  "db.m4.large", "db.m4.xlarge", "db.m4.2xlarge", "db.m4.4xlarge", "db.m4.10xlarge", "db.m4.16xlarge"
]

T Class General Purpose DB Instance Types

[
  "db.t3.micro", "db.t3.small", "db.t3.medium", "db.t3.large", "db.t3.xlarge", "db.t3.2xlarge",
  "db.t2.micro", "db.t2.small", "db.t2.medium", "db.t2.large", "db.t2.xlarge", "db.t2.2xlarge"
]

M Class General Purpose DB Instance Types

[
  "db.m5.24xlarge",
  "db.m4.large", "db.m4.xlarge", "db.m4.2xlarge", "db.m4.4xlarge", "db.m4.10xlarge", "db.m4.16xlarge"
]

Memory Optimized

[
  "db.r5.large", "db.r5.xlarge", "db.r5.2xlarge", "db.r5.4xlarge", "db.r5.12xlarge", "db.r5.24xlarge", 
  "db.r4.large", "db.r4.xlarge", "db.r4.2xlarge", "db.r4.4xlarge", "db.r4.8xlarge", "db.r4.16xlarge", 
  "db.x1e.xlarge", "db.x1e.2xlarge", "db.x1e.4xlarge", "db.x1e.8xlarge", "db.x1e.16xlarge", "db.x1e.32xlarge",
  "db.x1.16xlarge", "db.x1.32xlarge",
  "db.z1d.large", "db.z1d.xlarge", "db.z1d.2xlarge", "db.z1d.3xlarge", "db.z1d.6xlarge", "db.z1d.12xlarge"
]

R Class Memory Optimized DB Instance Types

[
  "db.r5.large", "db.r5.xlarge", "db.r5.2xlarge", "db.r5.4xlarge", "db.r5.12xlarge", "db.r5.24xlarge", 
  "db.r4.large", "db.r4.xlarge", "db.r4.2xlarge", "db.r4.4xlarge", "db.r4.8xlarge", "db.r4.16xlarge"
]

X Class Memory Optimized DB Instance Types

[
  "db.x1e.xlarge", "db.x1e.2xlarge", "db.x1e.4xlarge", "db.x1e.8xlarge", "db.x1e.16xlarge", "db.x1e.32xlarge",
  "db.x1.16xlarge", "db.x1.32xlarge"
]

Z Class Memory Optimized DB Instance Types

[
  "db.z1d.large", "db.z1d.xlarge", "db.z1d.2xlarge", "db.z1d.3xlarge", "db.z1d.6xlarge", "db.z1d.12xlarge",
]

When creating AWS CloudFormation templates that have EC2 instance type parameters, it can be tedious to create the allowed values list to validate the instance type string.

Here are some sets of instance types that can be copy-pasted into the "AllowedValues" field of your AWS CloudFormation EC2 instance type parameter.

View more info on these instance types: Amazon EC2 Instance Types

My Most Commonly Used Set

[
  "t3.2xlarge", "t3.large", "t3.medium", "t3.micro", "t3.nano", "t3.small", "t3.xlarge",
  "m5.12xlarge", "m5.16xlarge", "m5.24xlarge", "m5.2xlarge", "m5.4xlarge", "m5.8xlarge", "m5.large", "m5.xlarge"
]

All Instance Types

[
  "a1.2xlarge", "a1.4xlarge", "a1.large", "a1.medium", "a1.metal", "a1.xlarge",
  "c1.medium", "c1.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c3.large", "c3.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "c4.large", "c4.xlarge", "c5.12xlarge", "c5.18xlarge", "c5.24xlarge", "c5.2xlarge", "c5.4xlarge", "c5.9xlarge", "c5.large", "c5.metal", "c5.xlarge", "c5d.12xlarge", "c5d.18xlarge", "c5d.24xlarge", "c5d.2xlarge", "c5d.4xlarge", "c5d.9xlarge", "c5d.large", "c5d.metal", "c5d.xlarge", "c5n.18xlarge", "c5n.2xlarge", "c5n.4xlarge", "c5n.9xlarge", "c5n.large", "c5n.xlarge",
  "cc1.4xlarge", "cc2.8xlarge", "cg1.4xlarge", "cr1.8xlarge",
  "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "d2.xlarge",
  "f1.16xlarge", "f1.2xlarge", "f1.4xlarge",
  "g2.2xlarge", "g2.8xlarge", "g3.16xlarge", "g3.4xlarge", "g3.8xlarge", "g3s.xlarge", "g4dn.12xlarge", "g4dn.16xlarge", "g4dn.2xlarge", "g4dn.4xlarge", "g4dn.8xlarge", "g4dn.xlarge",
  "h1.16xlarge", "h1.2xlarge", "h1.4xlarge", "h1.8xlarge", "hi1.4xlarge", "hs1.8xlarge",
  "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "i2.xlarge", "i3.16xlarge", "i3.2xlarge", "i3.4xlarge", "i3.8xlarge", "i3.large", "i3.metal", "i3.xlarge", "i3en.12xlarge", "i3en.24xlarge", "i3en.2xlarge", "i3en.3xlarge", "i3en.6xlarge", "i3en.large", "i3en.metal", "i3en.xlarge",
  "inf1.24xlarge", "inf1.2xlarge", "inf1.6xlarge", "inf1.xlarge",
  "m1.large", "m1.medium", "m1.small", "m1.xlarge", "m2.2xlarge", "m2.4xlarge", "m2.xlarge", "m3.2xlarge", "m3.large", "m3.medium", "m3.xlarge", "m4.10xlarge", "m4.16xlarge", "m4.2xlarge", "m4.4xlarge", "m4.large", "m4.xlarge", "m5.12xlarge", "m5.16xlarge", "m5.24xlarge", "m5.2xlarge", "m5.4xlarge", "m5.8xlarge", "m5.large", "m5.metal", "m5.xlarge", "m5a.12xlarge", "m5a.16xlarge", "m5a.24xlarge", "m5a.2xlarge", "m5a.4xlarge", "m5a.8xlarge", "m5a.large", "m5a.xlarge", "m5ad.12xlarge", "m5ad.16xlarge", "m5ad.24xlarge", "m5ad.2xlarge", "m5ad.4xlarge", "m5ad.8xlarge", "m5ad.large", "m5ad.xlarge", "m5d.12xlarge", "m5d.16xlarge", "m5d.24xlarge", "m5d.2xlarge", "m5d.4xlarge", "m5d.8xlarge", "m5d.large", "m5d.metal", "m5d.xlarge", "m5dn.12xlarge", "m5dn.16xlarge", "m5dn.24xlarge", "m5dn.2xlarge", "m5dn.4xlarge", "m5dn.8xlarge", "m5dn.large", "m5dn.xlarge", "m5n.12xlarge", "m5n.16xlarge", "m5n.24xlarge", "m5n.2xlarge", "m5n.4xlarge", "m5n.8xlarge", "m5n.large", "m5n.xlarge",
  "p2.16xlarge", "p2.8xlarge", "p2.xlarge", "p3.16xlarge", "p3.2xlarge", "p3.8xlarge", "p3dn.24xlarge",
  "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "r3.large", "r3.xlarge", "r4.16xlarge", "r4.2xlarge", "r4.4xlarge", "r4.8xlarge", "r4.large", "r4.xlarge", "r5.12xlarge", "r5.16xlarge", "r5.24xlarge", "r5.2xlarge", "r5.4xlarge", "r5.8xlarge", "r5.large", "r5.metal", "r5.xlarge", "r5a.12xlarge", "r5a.16xlarge", "r5a.24xlarge", "r5a.2xlarge", "r5a.4xlarge", "r5a.8xlarge", "r5a.large", "r5a.xlarge", "r5ad.12xlarge", "r5ad.16xlarge", "r5ad.24xlarge", "r5ad.2xlarge", "r5ad.4xlarge", "r5ad.8xlarge", "r5ad.large", "r5ad.xlarge", "r5d.12xlarge", "r5d.16xlarge", "r5d.24xlarge", "r5d.2xlarge", "r5d.4xlarge", "r5d.8xlarge", "r5d.large", "r5d.metal", "r5d.xlarge", "r5dn.12xlarge", "r5dn.16xlarge", "r5dn.24xlarge", "r5dn.2xlarge", "r5dn.4xlarge", "r5dn.8xlarge", "r5dn.large", "r5dn.xlarge", "r5n.12xlarge", "r5n.16xlarge", "r5n.24xlarge", "r5n.2xlarge", "r5n.4xlarge", "r5n.8xlarge", "r5n.large", "r5n.xlarge",
  "t1.micro", "t2.2xlarge", "t2.large", "t2.medium", "t2.micro", "t2.nano", "t2.small", "t2.xlarge", "t3.2xlarge", "t3.large", "t3.medium", "t3.micro", "t3.nano", "t3.small", "t3.xlarge", "t3a.2xlarge", "t3a.large", "t3a.medium", "t3a.micro", "t3a.nano", "t3a.small", "t3a.xlarge",
  "u-12tb1.metal", "u-18tb1.metal", "u-24tb1.metal", "u-6tb1.metal", "u-9tb1.metal", "x1.16xlarge", "x1.32xlarge", "x1e.16xlarge", "x1e.2xlarge", "x1e.32xlarge", "x1e.4xlarge", "x1e.8xlarge", "x1e.xlarge", "z1d.12xlarge", "z1d.2xlarge", "z1d.3xlarge", "z1d.6xlarge", "z1d.large", "z1d.metal", "z1d.xlarge"
]

General Purpose

All General Purpose Types

[
  "a1.2xlarge", "a1.4xlarge", "a1.large", "a1.medium", "a1.metal", "a1.xlarge",
  "t1.micro", "t2.2xlarge", "t2.large", "t2.medium", "t2.micro", "t2.nano", "t2.small", "t2.xlarge", "t3.2xlarge", "t3.large", "t3.medium", "t3.micro", "t3.nano", "t3.small", "t3.xlarge", "t3a.2xlarge", "t3a.large", "t3a.medium", "t3a.micro", "t3a.nano", "t3a.small", "t3a.xlarge",
  "m1.large", "m1.medium", "m1.small", "m1.xlarge", "m2.2xlarge", "m2.4xlarge", "m2.xlarge", "m3.2xlarge", "m3.large", "m3.medium", "m3.xlarge", "m4.10xlarge", "m4.16xlarge", "m4.2xlarge", "m4.4xlarge", "m4.large", "m4.xlarge", "m5.12xlarge", "m5.16xlarge", "m5.24xlarge", "m5.2xlarge", "m5.4xlarge", "m5.8xlarge", "m5.large", "m5.metal", "m5.xlarge", "m5a.12xlarge", "m5a.16xlarge", "m5a.24xlarge", "m5a.2xlarge", "m5a.4xlarge", "m5a.8xlarge", "m5a.large", "m5a.xlarge", "m5ad.12xlarge", "m5ad.16xlarge", "m5ad.24xlarge", "m5ad.2xlarge", "m5ad.4xlarge", "m5ad.8xlarge", "m5ad.large", "m5ad.xlarge", "m5d.12xlarge", "m5d.16xlarge", "m5d.24xlarge", "m5d.2xlarge", "m5d.4xlarge", "m5d.8xlarge", "m5d.large", "m5d.metal", "m5d.xlarge", "m5dn.12xlarge", "m5dn.16xlarge", "m5dn.24xlarge", "m5dn.2xlarge", "m5dn.4xlarge", "m5dn.8xlarge", "m5dn.large", "m5dn.xlarge", "m5n.12xlarge", "m5n.16xlarge", "m5n.24xlarge", "m5n.2xlarge", "m5n.4xlarge", "m5n.8xlarge", "m5n.large", "m5n.xlarge"
]

A Class Arm-Based Types

[
  "a1.2xlarge", "a1.4xlarge", "a1.large", "a1.medium", "a1.metal", "a1.xlarge"
]

T Class General Purpose Types

[
  "t1.micro",
  "t2.2xlarge", "t2.large", "t2.medium", "t2.micro", "t2.nano", "t2.small", "t2.xlarge",
  "t3.2xlarge", "t3.large", "t3.medium", "t3.micro", "t3.nano", "t3.small", "t3.xlarge",
  "t3a.2xlarge", "t3a.large", "t3a.medium", "t3a.micro", "t3a.nano", "t3a.small", "t3a.xlarge"
]

M Class General Purpose Types

[
  "m1.large", "m1.medium", "m1.small", "m1.xlarge",
  "m2.2xlarge", "m2.4xlarge", "m2.xlarge",
  "m3.2xlarge", "m3.large", "m3.medium", "m3.xlarge",
  "m4.10xlarge", "m4.16xlarge", "m4.2xlarge", "m4.4xlarge", "m4.large", "m4.xlarge",
  "m5.12xlarge", "m5.16xlarge", "m5.24xlarge", "m5.2xlarge", "m5.4xlarge", "m5.8xlarge", "m5.large", "m5.metal", "m5.xlarge",
  "m5a.12xlarge", "m5a.16xlarge", "m5a.24xlarge", "m5a.2xlarge", "m5a.4xlarge", "m5a.8xlarge", "m5a.large", "m5a.xlarge",
  "m5ad.12xlarge", "m5ad.16xlarge", "m5ad.24xlarge", "m5ad.2xlarge", "m5ad.4xlarge", "m5ad.8xlarge", "m5ad.large", "m5ad.xlarge",
  "m5d.12xlarge", "m5d.16xlarge", "m5d.24xlarge", "m5d.2xlarge", "m5d.4xlarge", "m5d.8xlarge", "m5d.large", "m5d.metal", "m5d.xlarge",
  "m5dn.12xlarge", "m5dn.16xlarge", "m5dn.24xlarge", "m5dn.2xlarge", "m5dn.4xlarge", "m5dn.8xlarge", "m5dn.large", "m5dn.xlarge",
  "m5n.12xlarge", "m5n.16xlarge", "m5n.24xlarge", "m5n.2xlarge", "m5n.4xlarge", "m5n.8xlarge", "m5n.large", "m5n.xlarge"
]

Compute Optimized

[
  "c1.medium", "c1.xlarge",
  "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c3.large", "c3.xlarge",
  "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "c4.large", "c4.xlarge",
  "c5.12xlarge", "c5.18xlarge", "c5.24xlarge", "c5.2xlarge", "c5.4xlarge", "c5.9xlarge", "c5.large", "c5.metal", "c5.xlarge",
  "c5d.12xlarge", "c5d.18xlarge", "c5d.24xlarge", "c5d.2xlarge", "c5d.4xlarge", "c5d.9xlarge", "c5d.large", "c5d.metal", "c5d.xlarge",
  "c5n.18xlarge", "c5n.2xlarge", "c5n.4xlarge", "c5n.9xlarge", "c5n.large", "c5n.xlarge",
  "cc1.4xlarge", "cc2.8xlarge", "cg1.4xlarge", "cr1.8xlarge"
]

Memory Optimized

R Class Memory Optimized Types

[
  "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "r3.large", "r3.xlarge",
  "r4.16xlarge", "r4.2xlarge", "r4.4xlarge", "r4.8xlarge", "r4.large", "r4.xlarge",
  "r5.12xlarge", "r5.16xlarge", "r5.24xlarge", "r5.2xlarge", "r5.4xlarge", "r5.8xlarge", "r5.large", "r5.metal", "r5.xlarge",
  "r5a.12xlarge", "r5a.16xlarge", "r5a.24xlarge", "r5a.2xlarge", "r5a.4xlarge", "r5a.8xlarge", "r5a.large", "r5a.xlarge",
  "r5ad.12xlarge", "r5ad.16xlarge", "r5ad.24xlarge", "r5ad.2xlarge", "r5ad.4xlarge", "r5ad.8xlarge", "r5ad.large", "r5ad.xlarge",
  "r5d.12xlarge", "r5d.16xlarge", "r5d.24xlarge", "r5d.2xlarge", "r5d.4xlarge", "r5d.8xlarge", "r5d.large", "r5d.metal", "r5d.xlarge",
  "r5dn.12xlarge", "r5dn.16xlarge", "r5dn.24xlarge", "r5dn.2xlarge", "r5dn.4xlarge", "r5dn.8xlarge", "r5dn.large", "r5dn.xlarge",
  "r5n.12xlarge", "r5n.16xlarge", "r5n.24xlarge", "r5n.2xlarge", "r5n.4xlarge", "r5n.8xlarge", "r5n.large", "r5n.xlarge"
]

X Class Memory Optimized Types

[
  "x1.16xlarge", "x1.32xlarge",
  "x1e.16xlarge", "x1e.2xlarge", "x1e.32xlarge", "x1e.4xlarge", "x1e.8xlarge", "x1e.xlarge"
]

High Memory Class Memory Optimized Types

[
  "u-12tb1.metal", "u-18tb1.metal", "u-24tb1.metal", "u-6tb1.metal", "u-9tb1.metal"
]

Z Class Memory Optimized Types

[
  "z1d.12xlarge", "z1d.2xlarge", "z1d.3xlarge", "z1d.6xlarge", "z1d.large", "z1d.metal", "z1d.xlarge"
]

Accelerated Computing

P Class Accelerated Computing General Purpose GPU

[
  "p2.16xlarge", "p2.8xlarge", "p2.xlarge",
  "p3.16xlarge", "p3.2xlarge", "p3.8xlarge",
  "p3dn.24xlarge"
]

INF Class Accelerated Machine Learning Inference

[
  "inf1.24xlarge", "inf1.2xlarge", "inf1.6xlarge", "inf1.xlarge"
]

G Class Accelerated Computing GPU

[
  "g2.2xlarge", "g2.8xlarge",
  "g3.16xlarge", "g3.4xlarge", "g3.8xlarge", "g3s.xlarge",
  "g4dn.12xlarge", "g4dn.16xlarge", "g4dn.2xlarge", "g4dn.4xlarge", "g4dn.8xlarge", "g4dn.xlarge"
]

F Class Accelerated Computing FPGA Types

[
  "f1.16xlarge", "f1.2xlarge", "f1.4xlarge"
]

Storage Optimized

I Class Storage Optimized SSD-Backed Types

[
  "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "i2.xlarge",
  "i3.16xlarge", "i3.2xlarge", "i3.4xlarge", "i3.8xlarge", "i3.large", "i3.metal", "i3.xlarge",
  "i3en.12xlarge", "i3en.24xlarge", "i3en.2xlarge", "i3en.3xlarge", "i3en.6xlarge", "i3en.large", "i3en.metal", "i3en.xlarge"
]

D Class Storage Optimized HDD-Backed Types

[
  "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "d2.xlarge"
]

H Class Storage Optimized HDD-Backed Types

[
  "h1.16xlarge", "h1.2xlarge", "h1.4xlarge", "h1.8xlarge", "hi1.4xlarge", "hs1.8xlarge"
]

Colors in nature are often the result of pigmentation where the observed color is the result of selective absorption and reflection by the material. Objects that appear a certain color are so because they absorb all other colors in the spectrum, and reflect the one seen. Another form of color generation is not a result of pigmentation but is inherent of the material. Examples of this are peacock feathers, pearls, and wings of the Morpho [1]. These brilliant colors glisten with high reflectivity and vibrancy due to diffraction and reflection of light at the microscale [1].

Structural colors have special properties such as intensity patterns under different orientations, and brilliant sparkling or glistening on the surface. This phenomenon has been studied extensively and the papers referenced here support a multi-layered model seen in figure 4. The tree-like structures reflect the light and cause constructive and destructive interference at the micro level in the material [8].

This paper will survey a replication technique using soft lithography, identify challenges and propose an alternative. The method will replicate the microstructure of the Morpho butterfly wing using a layered deposition approach from [6]. Improvements and methods are proposed to broaden the scope of the experiment in [6] to include more critical features of the wing's microstructure.

Introduction

The Morpho butterfly wing is an example of a structural color that has high reflectivity in a certain wavelength range and uniform reflection in a large angular range. Seen in figure 2 are the ridges and valleys of the wing that resulted in the color and speckles on the surface. It has been shown that the varying heights of the ridges result in the sparkles on the surface. The ridges act as scatterers of the incoherent light and the air between diffracts the light causing constructive and destructive interference [1].

High reflectivity is obtained from the multiple layers of the lamella and their separation causing multiple reflections [2]. Note the pigment at the base of the lamella in figure 4, its purpose is to absorb other colors in the spectrum enhancing the electric blue. Another important feature of the microstructure is the fact that the ridge heights are randomly distributed. This is supported experimentally in [2], as there were no diffraction spots observed which would be an indicator of uniform ridge heights.

The complete process of producing the structural color of the Morpho wing is outlined and shown in figure 4. As the incident light penetrates the material, it is scattered as it passes through the multiple layers of the lamella. The ridges scatter the light into the gaps where air diffracts and interferes with itself, finally being reflected back out of the material. The color observed is due to the constructive interference as it leaves the material. When the angle of the incident light changes, the observed color changes due to the different interference path through the material [1], 8.

Although a replica was not created in [2], their proposed model is in alignment with many of the observations of the natural Morpho wing such as the angular dependency on reflection intensity.

The results from [2] support interference within a single lamella. There is uniform diffusion due to the random distribution of ridge heights which eliminate diffraction spots. High reflectivity is achieved by the small separation between lamella and their compact tree structure, and finally, the pigment at the base of the lamella enhances the blue color by absorbing all other colors [2].

Replicating Microstructures Through Soft Lithography

Many studies have been carried out on the mechanisms used in the structural color of the Morpho wing. The experiment discussed here attempts to replicate the microstructure using molding lithography. The techniques and results in [7] are included to survey their outcomes and explore other possible methods in replicating the microstructures. The experiment in [7] employed soft lithography to create a polydimethylsiloxane (PDMS) replica of the wing.

Details of the steps and the schematic illustration of the process can be found in [7], but the basic steps included pouring a PDMS solution on the wing, curing, lift off of the mold and the final, sputtering a layer of Pt/Au on the surface of the replica. This layer emulates the natural surface finish of the wing and will cause the required scattering [7]. The physical dimensions of the replica were measured and compared to those of the original wing as well as absorptivity and reflectivity under varying incident wavelengths [7].

Results and Conclusions on Replica Quality and Resolution

The significant difference in scale between the replica and natural wing is due to lack of penetration of the PDMS solution into the lamellar structures of figure 3. The residue left during curing is also responsible for increasing the dimensions of the replica [7].

Ultimately the replica's scale was larger, contained less tree-like lamellar microstructures but because of the added layer of Pt/Au, actually had higher surface reflectivity [7]. Wetting issues may also cause lack of resolution as the wing surface is hydrophobic. This observation was documented in [7] which allows the wing to function even in wet conditions. The soft lithography technique was positively regarded as it still produced microstructures with reasonable accuracy as seen in figure 1.

Alternating Multilayer Deposition Proposal

Observe in figure 4 the cross-section of a lamellar structure. These structures are patterned across the surface of the wing and down to the cuticle. These structures are simple in geometry but their position and orientation limit replicating techniques.

The method in [7] used the wing as a positive cast for the PDMS mold. This will inevitably result in a larger model than the natural wing. As stated, the resolution was also limited due to lack of penetration of the solution.

The method proposed is a layered approach taken in [6]. One of the major drawbacks of the method in [7] was the lack of resolution of the tree-like structures. These features are critical in the diffraction and interferences required for color generation. The methods in [6] are motivated by benefits in electronics and displays and chosen here for its feasibility and flexibility in the parameters.

The method is outlined in figure 5 and uses alternating layers of $Si_{3}N_{4}$ and $SiO_{2}$ deposited on a $Si$ substrate. This allows for anisotropic and isotropic etching used to replicate the 3D structures seen in figure 2. This method is relatively simple and makes use of multilayer deposition which is more feasible than other methods such as focused ion beam Chemical Vapor Deposition (CVD) [6].

The focus on the tree-like structures is due to the fact that the color and reflective properties are generated from the multiple reflections and diffractions within each lamella. Therefore this structure is most important when the goal is color and reflective accuracy.

The initial experiment in [6] resulted in a wafer that had very low reflectivity. This was claimed to be due to the tapered edges of the ridges caused by the isotropic etch and would require improvement of the etching strategy to decrease damage [6]. The color was still visible in spite of the low reflectivity.

Improving the Layered Approach

Improving reflectivity is carried out by reversing the stack order and starting with $SiO_{2}$ on the substrate. High reflectivity was achieved by constructive interference between the reflected light of the nitride-oxide interface and the oxide-substrate interface. This stacking order also increased the number of layers as the top layer was no longer isotropically etched [6].

Using this method, many properties of the replica can be varied such as gap thickness, lamella thickness and width, and dialectic material in the gap 6. In the natural wing, the width of the air gap is such that it maximizes reflectivity but this feature is not trivial to replicate [6].

The natural wing also has irregularities that contribute to the diffraction and color. The study in 6 purposely did not recreate these irregular features. Including the random distribution in the lamellae, heights could be achieved by starting with a random uneven substrate. This could be produced by first etching the substrate with a randomly generated mask prior to layering the oxides and nitrides.

The asymmetry of a single lamella is a challenge to replicate, but some of the work has been done by the isometric etch. Observe in figure 6 the irregular pattern of the tree-like branches of the lamella, these are not perfectly square edged features. The isotropic etch in fact improves the accuracy by tapering the edges.

The cross-sectional view of the lamellae in figure 6 appear conical from the base to tip. Replicating this feature could be achieved by using a partial anisotropic etch during step 5in figure 5 by rotating the mask to expose the $[111]$ plane.

Since isotropic etching is agitative sensitive, this can be leveraged to create a more irregular pattern through uneven agitation during the isotropic etch. Conversely, since anisotropic etching is reaction rate limited, this process could be timed so as not to over-etch the top layers of the lamella. This is normally a challenge when the goal is creating highly uniform features but is a great asset when an irregularity is a goal.

Conclusion

Colors in nature are often the result of pigmentation where certain wavelengths in the spectrum are absorbed and the one observed is reflected. Structural colors are a phenomenon created through purely physical means. The Morpho butterfly wing is an example of a vibrant blue, highly reflective and dynamic structural color. The source of this color is caused by multiple reflections and diffractions within the microstructure. This constructive and destructive interference creates the emitting color.

The method proposed to replicate the microstructure of the wing is a layered approach given in [6]. The scope of the experiment did not include replicating the irregularities but rather focused on the tree-like structure of the lamella. These features are important for causing the diffractions and reflections that create the resulting color.

Improvements were proposed such as randomly etching the substrate prior to layering in order to replicate the random distribution of the ridge heights. In addition, the creative use of anisotropic and isotropic etching can help in replicating the cone shape and irregularities of the inner tree-like structures.

Including irregularities of the ridges and the tree-like structure was shown to produce more accurate results and the proposed method aims at reproducing these critical features, optimized for the replica's materials and scales 1, 2, 6.

References

1. Shuichi Kinoshita, Shinya Yoshioka and Kenji Kawagoe, ``Mechanisms of structural colour in the Morpho butterfly: cooperation of regularity and irregularity in an iridescent scale," The Royal Society, Froc. R. Soc. Lond. B 269, 2002
2. Shuichi Kinoshita, Shinya Yoshioka, Yasuhiro Fujii and Naoko Okamoto, ``Photophysics of Structural Color in the Morpho Butterflies," Forma, 17, 103–121, 2002
3. T. L. Tan, D. Wong, and P. Lee, ``Iridescence of a shell of mollusk Haliotis Glabra," Vol. 12, No. 20, Optics Express 4847, 2004
4. P. Vukusic, J. R. Sambles, C. R. Lawrence and R. J. Wootton, ``Quantified interference and diffraction in single Morpho butterfly scales," Proc. R. Soc. Lond. B 266, 1403-1411, 1999
5. Zhigang Chen and Allen Taflove, ``Superenhanced backscattering of light by nanoparticles," Optics Letters, Vol. 31, No. 2, 2006
6. Rebecca E. Coath, ``Investigating the Use of Replica Morpho Butterfly Scales for Colour Displays," University Of Southampton, School Of Electronics And Computer Science, 2007
7. Shao-Hui Kang, Tzu-Yao Tai, Te-Hua Fang, ``Replication of butterfly wing microstructures using molding lithography," Current Applied Physics 10, 625–630, 2010
8. Andrew R. Parker, David R. McKenzie and Maryanne C. J. Large, ``Multilayer reflectors in animals using green and gold beetles as contrasting examples," The Journal of Experimental Biology 201, 1307–1313, 1998

If you're using Vim Airline, there are many beautiful themes to pick from. But I personally like to remove the noise and use color sparingly.

This theme has status line background color and no bright highlights in normal mode. When color is used, it's minimal and only highlights some (arguably) important information.

Edit (January 27, 2020): I was recently asked by Rishabh Mehta about the details of my setup to achieve this look. Here it is:

• Font: Fira Code https://github.com/tonsky/FiraCode
• Terminal vim (not gvim). Currently using NeoVim and works just the same.
• At the time of this post, I was using iTerm2 (https://iterm2.com/) with 165% line height. Currently I'm using kitty terminal (https://sw.kovidgoyal.net/kitty/) with 150% line height.
• Color scheme: Gruvbox (https://github.com/morhetz/gruvbox). Although I rotate between my absolute favorite: Ayu Mirage (https://github.com/ayu-theme/ayu-vim) and my latest Photon (https://github.com/axvr/photon.vim)

Normal and Normal Modified Mode

Normal mode has no highlighting, no background colors. The normal mode orange is used to indicate unsaved changes.

Insert and Visual Mode

Insert mode is green for go!

Replace and Select Mode

Active and Inactive Split

Removed much of the information from the inactive pane when using splits. This keeps the focus on the active pane without adding color.

Getting The Theme

I don't have a name for it, currently it's temporarily called transparent.vim, eventually I'll make a pull request but for now, you can copy the contents below.

NeoVim
~/.local/share/nvim/plugged/vim-airline-themes/autoload/airline/themes/transparent.vim

Vim
~/.vim/plugged/vim-airline-themes/autoload/airline/themes/transparent.vim

Edit (July 16, 2020): For completion, João Paulo Pesce sent me a snippet to colorize Airline errors, see added section at the bottom. I tested with Syntastic and it looks much better :)

" Colors
let s:gray     = [245, '#3f4b59']
let s:darkgray = [245, '#1d1f21']
let s:golden   = [143, '#BBE67E']
let s:pink     = [131, '#F07178']
let s:blue     = [ 67, '#D4BFFF']
let s:orange   = [166, '#ffae57']
let s:outerfg  = [ 16, '#8d96a1']
let s:outerfgi = [ 16, '#2f3d4d']

" Backgrounds
let s:outerbg  = [ 16, 'NONE']
let s:innerbg  = [234, 'NONE']

" Normal mode
let s:N1 = [s:outerfg[1], s:outerbg[1], s:outerfg[0], s:gray[0]]
let s:N3 = [s:gray[1]   , s:innerbg[1], s:gray[0]   , s:innerbg[0]]

" Normal mode - modified
let s:NM1 = [s:darkgray[1], s:orange[1], s:darkgray[0], s:orange[0]]
let s:NM3 = [s:orange[1]  , s:outerbg[1], s:orange[0], s:darkgray[0]]

" Insert mode
let s:I1 = [s:darkgray[1], s:golden[1], s:outerfg[0], s:golden[0]]
let s:I3 = [s:golden[1]  , s:innerbg[1], s:golden[0], s:innerbg[0]]

" Visual mode
let s:V1 = [s:darkgray[1], s:pink[1], s:outerfg[0], s:pink[0]]
let s:V3 = [s:pink[1]    , s:innerbg[1], s:pink[0], s:innerbg[0]]

" Replace mode
let s:R1 = [s:darkgray[1], s:blue[1], s:outerfg[0], s:blue[0]]
let s:R3 = [s:blue[1], s:innerbg[1], s:blue[0], s:innerbg[0]]

" Inactive pane
let s:IA = [s:darkgray[1], s:outerbg[1], s:innerbg[0], s:innerbg[0]]
let s:IAc = [s:gray[1], s:outerbg[1], s:outerbg[0], s:outerbg[0]]

let g:airline#themes#transparent#palette = {}
let g:airline#themes#transparent#palette.accents = {
    \ 'red': ['#d70000', '', 160, '', '']}

let g:airline#themes#transparent#palette.inactive = {
    \ 'airline_a': s:IA,
    \ 'airline_b': s:IA,
    \ 'airline_c': s:IAc,
    \ 'airline_x': s:IA,
    \ 'airline_y': s:IA,
    \ 'airline_z': s:IA}

let g:airline#themes#transparent#palette.inactive_modified = {
    \ 'airline_a': s:IA,
    \ 'airline_b': s:IA,
    \ 'airline_c': s:NM3,
    \ 'airline_x': s:IA,
    \ 'airline_y': s:IA,
    \ 'airline_z': s:IA}

let g:airline#themes#transparent#palette.normal = {
    \ 'airline_a': s:N1,
    \ 'airline_b': s:N3,
    \ 'airline_c': s:N3,
    \ 'airline_x': s:N3,
    \ 'airline_y': s:N3,
    \ 'airline_z': s:N3}

let g:airline#themes#transparent#palette.normal_modified = {
    \ 'airline_a': s:NM1,
    \ 'airline_b': s:N3,
    \ 'airline_c': s:N3,
    \ 'airline_x': s:N3,
    \ 'airline_y': s:N3,
    \ 'airline_z': s:NM3}

let g:airline#themes#transparent#palette.insert = {
    \ 'airline_a': s:I1,
    \ 'airline_b': s:N3,
    \ 'airline_c': s:N3,
    \ 'airline_x': s:N3,
    \ 'airline_y': s:N3,
    \ 'airline_z': s:I3}
let g:airline#themes#transparent#palette.insert_modified = {}

let g:airline#themes#transparent#palette.replace = {
    \ 'airline_a': s:R1,
    \ 'airline_b': s:N3,
    \ 'airline_c': s:N3,
    \ 'airline_x': s:N3,
    \ 'airline_y': s:N3,
    \ 'airline_z': s:R3}
let g:airline#themes#transparent#palette.replace_modified = {}

let g:airline#themes#transparent#palette.visual = {
    \ 'airline_a': s:V1,
    \ 'airline_b': s:N3,
    \ 'airline_c': s:N3,
    \ 'airline_x': s:N3,
    \ 'airline_y': s:N3,
    \ 'airline_z': s:V3}
let g:airline#themes#transparent#palette.visual_modified = {}


" Warnings
let g:airline#themes#transparent#palette.normal.airline_warning = s:NM1

let g:airline#themes#transparent#palette.normal_modified.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning

let g:airline#themes#transparent#palette.insert.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning

let g:airline#themes#transparent#palette.insert_modified.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning

let g:airline#themes#transparent#palette.visual.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning

let g:airline#themes#transparent#palette.visual_modified.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning

let g:airline#themes#transparent#palette.replace.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning

let g:airline#themes#transparent#palette.replace_modified.airline_warning =
    \ g:airline#themes#transparent#palette.normal.airline_warning


" Errors
let g:airline#themes#transparent#palette.normal.airline_error = s:V1

let g:airline#themes#transparent#palette.normal_modified.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error

let g:airline#themes#transparent#palette.insert.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error

let g:airline#themes#transparent#palette.insert_modified.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error

let g:airline#themes#transparent#palette.insert_modified.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error
 
let g:airline#themes#transparent#palette.visual.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error

let g:airline#themes#transparent#palette.visual_modified.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error
 
let g:airline#themes#transparent#palette.replace.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error

let g:airline#themes#transparent#palette.replace_modified.airline_error =
      \ g:airline#themes#transparent#palette.normal.airline_error

About Me

I'm a software engineer and mechanical engineer. I studied control systems at the University of California, Irvine and hold a Masters degree in Mechanical and Aerospace Engineering. My broad background helps me quickly learn, adapt and stay passionate about finding pragmatic solutions to complicated problems.

Skip the online JSON formatters! Run this directly in the vim command line:

:!python -m json.tool

This is very handy when downloading or working with minified JSON and works in your current buffer.

To make it even easier, I set up a shortcut in my vimrc which will run the above command but also set the filetype to json.

nnoremap ,j :set ft=json<cr>:%!python -m json.tool<cr>gg=G<cr>

Note my map leader is , (comma), so ,j (for JSON) will call the function on my current buffer.

While some have characterized the wrist as a second order system, the focus here is fitting a more complex fourth order model that accurately represents observed responses and muscle activity [1].

Prior to the publishing of [1], most studies of the wrist did not include an explicit model when analyzing responses.

Using this model and Hill’s equation relating force and velocity with electromyograms (EMG), a controller can be designed that correctly activates the wrist muscles to achieve desired movements. Furthermore, characterizing the wrist using this model lends itself to a straightforward approach to designing a functional electrical stimulation (FES) system.

Terms and Definitions

Alpha motor neurons are responsible for direct activation of skeletal muscles. Motor units include the alpha motor neuron and all of the muscle fibers which it is responsible for activating. Muscle fiber is the strand of tissue that includes the myofibrils at its core, activated by the axon attached at its surface. Muscle fibril or myofibril makes up the core of the muscle fiber. Myofilaments make up the myofibril and are constructed of proteins. The thin filament is activated by the thick filament, myosin. Actin is a protein that makes up the thin filament. Myosin attaches and pulls on the actin producing contraction. Sarcomere surrounds each filament and anchors the myosin fibers.

Fused tetanus is a contraction that does not have oscillations. Unfused tetanus is a contraction with unsteady contractions resulting in oscillations. Recruitment refers to the order in which muscles are activated, using muscles that fatigue slower first and resorting to faster fatiguing muscles last. Size principle refers to the excitability of small motor units opposed to larger ones. As signal strength increases, large motor units are recruited, gradually increasing contractile force.

Figure 1 is the diagram for the muscle components. Observe that during contraction, the actin filaments move closer together reducing the H-Zone, and overlap during extreme conditions. This overlapping reduces the number of actin attachment sites to myosin. With less active area contract- ing, the force decreases.

The opposite condition during extreme extension causes the H-Zone to grow large enough that actin-myosin attachment sites decrease and in turn decrease force. This case reduces force more drastically than an extreme contraction where actin-myosin sites can still reattach during reduced contraction.

Linear Model Simulation

The objective is to measure isometric force produced by the wrist in response to a step input.  Two values of stiffness are used during an equal and opposite contraction of the flexor and extensor muscles.  The linear model is shown in figure 2 where $U_{e}$ and $U_{f}$ are the inputs.  Stiffening of the muscles is a common problem after spinal cord injuries and this simulation will compare the response due to increased muscle stiffness.

The equations of motion for the mass $m$ are as follows:

Where the forces are:

Since the damping terms are constant, we can define new variables and represent the dynamics as a set of first order equations. Let $x_{1} = x$, $x_{2} = \dot{x}$ and rearrange the above equations:

Simulation of the dynamics is straight forward, requiring only the state space representation of the model and the step input. The response time is measured when the isometric muscle force reaches 6.3 grams. Figure 3 shows force vs time at a stiffness of 500 g/cm. The time required to generate this force is listed on the datatap as 0.01989 sec.  When the stiffness is doubled in figure 4, the response time decreases, as expected, to 0.009943 sec.

The linear model with constant stiffness and damping is valid in a region around the equilibrium, but cannot be reliably extended to extreme cases or global stability. Nonlinear analysis in the next section will assume velocity and state dependent stiffness and damping.

Nonlinear Model Simulation

The objective is to move the wrist from an initial position at $10^\circ$ extension to a $20^\circ$ flexion as fast as possible.  Equations of motion are formulated from the model shown in figure 4, and will be used to simulate responses with different parameters.

Writing $F=ma$ for the mass $m$:

Where the forces are:

The viscosities $B_{f}$ and $B_{e}$ are not constant, they vary with velocity $\dot{x_{f}}$, $\dot{x_{e}}$ and active state $U_{f}$, $U{e}$. Using Hill's equation, the viscosities can be defined as follows and substituted back into the force equations.

Now we can define new variables and represent the dynamics as a set of first order equations. Let $x_{1} = x$, $x_{2} = \dot{x}$ and rearrange the above equations:

Where the stiffnesses are also dependent on active state and defined:

With the equations now in first order form, MATLAB can be used to integrate the dynamics and a control law designed with $U_{f}$ and $U_{e}$ as inputs and $x_{1}$ as output wrist angle. The variables $a$ and $b$ and control input are free parameters and can be chosen by trial and error or by optimizing a cost function.

Simulation and Optimization

The dynamics are simulated using Simulink where the equations are integrated simultaneously with a variety of input signals.  Trial and error approach is used to determine $a$ and $b$ knowing from equation 4 that increasing $a$ and decreasing $b$ will increase the viscous damping terms. Parameters of the input functions are also determined by systematically adjustment of amplitude, frequency, pulse width and the degree of interleaving.

The simulation and recorded results of the agonist and antagonist muscle reveal that a fast response depends on activity of both muscles at delayed intervals. Using this strategy to design the control inputs, square wave pulses are interleaved to increase overall stiffness and drive the wrist to the target. Fast responses resulted in oscillations, however by varying $a$ and $b$ to adjust the damping, a reasonable response is obtained.

The simulink model is shown in figure 6 below. Figures 7-8 show the wrist trajectory as it moves to the target and the velocity during the movement. Figures 9-10 show the agonist and antagonist muscle input signals. Figures 11-12 show the torques in the flexor and extensor muscles over the time period. Observe that the maximum torques in either muscle do not exceed the muscles' maximum torques.

Conclusions

Observe from figure 7, the time required to move from $1^\circ$ away from initial to within $1^\circ$ of the final target is $0.46 sec$. This is significantly slower compared to the results from [1] where the same response is executed in around $0.25 sec$. However, this simulation does not include a parameter optimization algorithm.

Using an explicit model to describe the wrist response allows for systematic design of an FES system.  Activating the agonist and antagonist muscles of the model produce a response very close to that of the actual hand [1]. Since muscle stimulations are explicitly known, they can be reconstructed and optimized for each patient.

As stated, the muscles of the wrist are accessible for EMG measurement and likewise for surface stimulation. Varieties of movements can be obtained by adjusting the stimulation parameters and/or the interleaving of signals. Though torques were below maximum levels, stimulation strength may exceed thresholds.  However, with a functional model, many nonlinear control strategies can be tested for advantages and customized for each patient.

MATLAB Code & Model

Below is the code to run the linear model.

% Linear Model Simulation of the Wrist
clear; close all; clc;

% Define the variables of the system as real numbers (symbolic)
syms x1 x2 xf xe x Uf Ue real

K = 500; % Stiffness
B = 10; % Damping
m = 500; % Mass

kf = K; ke = K;
Bf = B; Be = B;

t = 0:.0001:.4;

% State-Space Equations of Motion
Am(1,:) = [0 1 0 0];
Am(2,:) = [(1/m)*(-ke - kf) 0 -kf/m ke/m];
Am(3,:) = [-kf/Bf 0 -kf/Bf 0];
Am(4,:) = [ke/Be 0 0 -ke/Be];

U = [Uf Ue]';

Bm = [0 0; 0 0; 1/Bf 0; 0 1/Bf];

Cm = [1 0 1 0];
Dm = 0;

Sys = ss(Am, Bm, Cm, Dm);

[y,t,x] = step(10*Sys, t);

h = figure(1);
set(h, 'position',[500 500 600 250])
plot(t, y(:,1)*K); grid on;
title('Isometric Muscle Force vs. Time at K = 500 g/cm');
xlabel('Time (s)'); ylabel('Isometric Muscle Force (g)');
axis([0 0.15 0 12])

Download the MATLAB code and Simulink Model to try out the nonlinear Hill Model simulation.

References

1. S.L. Lehman and B.M. Calhoun, ``An Identified Model for Human Wrist Movements," Experimental BrainResearch, 1989

Including more information about the person's physiological state has been shown to affect BCI performance. Measurement of eye position can provide information about the general location of a desired movement.

Brain Computer Interfaces (BCI) measure brain activity and allow subjects to control a computer or robotic device purely through imagery. Accuracy of BCI devices vary greatly between subjects and for many, performance is not at a functional level [2].

Measurement noise and sensing technology are a few factors that contribute to the challenge of BCI decoding. Including more information about the person's physiological state has been shown to affect BCI performance [2]. In general when one reaches for or manipulates an object, the eyes are fixed upon it. Measurement of eye position can provide information about the general location of a desired movement. In addition, measurement of eye focus may provide information about depth perception [8]. Combining BCI readings for direction and filtering from eye position and pupil dilation may increase overall BCI performance.

Introduction

Measuring activity in the sensory motor cortex (SMC) is the basis of BCI function. This ability allows for translation of brain activity into specific commands for computers or actuators. Recording sensory motor rhythms (SMR) allow the BCI to extract information about the subject's intended movement and in turn target that location using a cursor or robotic device. Even though in general, people can voluntarily modulate their SMR [1], decoding accuracy is highly dependent on the subject and as a result requires many calibration trials [2].

Through muscle stimulation and robotics, BCIs have achieved complex movement capabilities relying purely on visual feedback [12]. When planning a movement, the brain uses visual information about the target and translates it into a desired trajectory that is used to execute the movement [11]. Because of the strong dependence on visual feedback, including physiological information of the eye can provide clues about intent of the subject [2].

Eye tracking devices allow patients with locked-in syndrome to communicate by tracking eye movement through infrared sensing. Video recording is used as well and processed by the software, providing a canvas which allows interaction exclusively through eye movement. Similar to virtual control in BCIs, eye tracking devices merely turn the eye into a cursor on a computer screen.

The devices do not measure eye focus, but there is a correlation between pupil size and focus distance [10]. Relative changes in pupil size can be recorded using video feedback already present on eye trackers. With the added measure of depth, a preliminary target in space can be established and used by the filtering algorithms during decoding.

The Motivation

As stated in [2], many papers will only include results for participants who successfully achieved BCI control. Many challenges are still present such as illiteracy, measurement noise, variability in performance and long training periods
[1, 2, 3, 4, 6, 11]. In fact, research has shown that 15-30% of participants experience
BCI illiteracy in which no control is achieved [2]. This phenomenon leads to long, frustrating training sessions that ultimately do not result in BCI control. Those that do achieve success, have training periods ranging from 30 minutes to 2 weeks
and movements are generally slow and unrefined [11].

Activity in the SMC has been observed as general commands for a motion that are more precisely carried out by the upper spinal cord [13]. Therefore, exclusive measurement of SMC activity may lack valuable information, resulting in poor BCI performance or illiteracy [13].

Research of BCI performance while monitoring neurophysiological signals has shown positive and negative effects depending on mood and motivation [2, 3, 6]. Variability between subjects was still significant in spite of including neurophysiological predictors in [2, 3]. Noise generated by occipital neural activity was detected during BCI use in [2]. Further experimentation revealed that idling of the vision lead to increased distortion of SMR measurement [2]. Interestingly, engagement of the visual senses in an activity reduced noise generated by occipital idling rhythms [2].

Motivational factors have a significant impact on BCI performance as well [3]. Subjects tasked with uninteresting or unchallenging goals experienced low BCI performance in [3]. When these subjects were given a simple simulated environment, performance was significantly better. Though still unknown, factors such as novel challenges and fun/interesting tasks have been associated with these results [3]. The neurophysiological responses observed in [2, 3] are valuable to the training paradigm and filtering strategy used in BCI design.

The Hypothesis

The focus of this paper is on physiological clues of a subject's intended movement by measure of pupil position and diameter. There is strong dependence on visual feedback during planning and execution of movements in able-bodied and BCI dependent subjects [2, 11]. Including physiological information of the eye may aid in the design of BCI filtering strategies.

Highly accurate and robust eye tracking devices have been developed that are low cost and noninvasive. These devices do not measure eye focus directly, but pupil size can be easily determined from live imaging of the eye. Correlation between relative
changes in pupil size and focus can provide a relative measure of target depth [8, 9, 10]. Tracking pupil position and focus would narrow the range of the intended motion while also providing a general target in the 3D space of the subject. Combining current BCI decoding strategies with information from eye position and depth perception can greatly improve performance. With this information, decoding algorithms can filter SMR activity more selectively and define a threshold around a preliminary target prediction.

The Technology

Planar Eye Tracking Devices

Devices such as the EyeWriter and GazeTracker are projects that provide open-source, publicly licensed software for eye tracking. The complete setup uses infrared LEDs and video recording that sense pupil position. The software filters and refines the data in order to track eye position and velocity [14, 16]. In addition to infrared sensing, video recording is processed using common web cam hardware [15, 16].

Both devices are noninvasive, the GazeTracker centralizes all its electronics and compensates for head movement. This device has even achieved EMG clicking capability for use with their software [14]. Both devices have successfully enabled patients with ALS to communicate and even produce artwork purely from eye movement [14, 15, 16].

Both devices use a computer interface which allow subjects to select letters and words in a text processor. The Eyewriter provides a canvas as well and its refined, smooth tracking capability allows its subjects to produce fine, detailed drawings.

Depth Measurement

Many environmental factors influence the physiological responses of the eye; limited here to the effects on pupil diameter [7, 8, 9]. Previous works have studied the effects of illumination, focus distance and mental alertness on pupil accommodation [9]. The objective here is asses the possibility of determining object distance based only on pupil diameter.

The experiment in [9] directly measured pupil diameter at systematic depth fixation intervals. Observe in figure 1 the monotonic increase in pupil size over the range of focus distance. The data suggests a linear relationship in the space from $10cm$ to $50cm$ from the body [9]. These results provide a straight forward method to calibrating and predicting depth fixation in patients using existing eye tracking devices.

Obtaining the data in [9] required development of software methods to measure pupil size from recorded images. These methods can be ported directly to the eye trackers discussed, in combination with the linear model defined in [9].

Integration

Current technology can track the eyes and project their motion on a 2D plane, typically a computer screen or wall projection. Measurement of pupil size can be determined from the existing video processing and changes in pupil size can be recorded. Depth calibration, in theory could be carried out with only 2 test points at different distances. With planar tracking and depth fixation distance, a general target in 3D space can be determined.

Due to the high noise in measurement of brain activity, BCIs typically include some of form of band-pass filtering, Laplace-filtered channels or subject optimized spacial filters [2]. Figure 2 depicts the population vector algorithm used in BCI decoding. The approach provides a means of measuring neurons and predicting the desired movement direction. Noise from neuron pools arise from a variety of uncontrollable sources making decoding very challenging [5]. As a result, the algorithm may converge on different targets for the same intended movement [5].

The approach is to use eye position and focus to define a general location in 3D space in front of the subject. Updating this prediction based on the subject's visual attention would filter neuron activity outside a threshold around the target. Eliminating extremes from the data pool may increase consistency of convergence. In addition, adapting the population vector algorithm to include a weighting based on visual feedback may allow for prefiltering of the data upfront.

With current eye tracking technology and the added dimension from depth measurement, a preliminary target prediction can take place before any BCI prediction takes place. Perhaps combining this with existing filtering techniques may decrease BCI illiteracy and increased overall performance for existing users.

Conclusion

For years BCIs have achieved success in performing complex movements such as gripping and reaching [12]. This technology allows subjects to control a computer or robotic device by imagining a movement [2]. After 30 years of research, BCI performance is still very sensitive to each subject and for many, does not meet criteria [2]. Including more information about the person's physiological state has been shown to increase BCI performance [1, 3].

Using eye tracking devices to measure planar motion and depth fixation would allow for a preliminary target definition in space. Establishing a threshold and weighting scheme around this target may significantly filter noise and increase accuracy and precision of BCI devices.

References

1. Fetz, E.E. ``Volitional control of neural activity: implications for brain computer interfaces," Journal of Physiology, Article 579, 571–579, 2007
2. Blankertz B. et al. ``Neurophysiological predictor of SMR-based BCI performance," NeuroImage, Article 51, 1303–1309, 2010
3. Nijboer F., Birbaumer N. and Kubler A. ``The influence of psychological state and motivation on brain–computer interface performance in patients with amyotrophic lateral sclerosis– a longitudinal study" Frontiers in Neuroscience, Vol. 4, Article 55, July 2010
4. Schwartz A.B. et al. ``Brain-controlled interfaces: movement restoration with neural prosthetics," Neuron, Article 52, 205–220, October 2006
5. Georgopoulos A. P. ``Translation of directional motor cortical commands to activation of muscles via spinal interneuronal systems" Cognitive Brain Research, 151-155, 1996
6. Kubler, A. et al. ``Patients with ALS can use sensorimotor rhythms to operate a brain computer interface," Neurology, Article 64, 1775–1777, 2005
7. Marzocchi N., Breveglieri R., Galletti C. and Fattori P. ``Reaching activity in parietal area V6A of macaque: eye influence on arm activity or retinocentric coding of reaching movements," European Journal of Neuroscience, Vol. 27, 775–789, 2008
8. Marcos S., Moreno E., Navarro R. ``The depth-of-field of the human eye from objective and subjective measurements," Vision Research, Article 39, 2039–2049, 1999
9. Lee E. C., Lee J. W., and Park K. R. ``Experimental Investigations of Pupil Accommodation Factors," IOVS Papers in Press, Manuscript iovs.10-6423, February 2011
10. Ripps H., Chin N. B., Siegel I. M., and Breinin G. M. ``The effect of pupil size on accommodation, convergence, and the AC/A ratio" Investigative Ophthalmology, Vol. 1, Article 1, February 1962
11. Green A. M. and Kalaska J. F. ``Learning to move machines with the mind" Trends in Neurosciences, Vol. 34, Article 2, February, 2011
12. O’Doherty J. E. ``Active tactile exploration using a brain machine brain interface" Nature, Vol. 479 228-332, November 2011
13. Dum, R.P. and Strick, P.L., ``The corticospinal system, a structural framework for the central control of movement," American Physiological Society, New York, 1996
14. San Agustin J., et al. ``Evaluation of a low-cost open-source gaze tracker." ACM, New York, NY, USA, 77-80, 2010. http://doi.acm.org/10.1145/1743666.1743685
15. San Agustin J., Hansen J. P., Hansen D. W. and Skovsgaard H. ``Low-cost gaze pointing and EMG clicking" ACM, New York, NY, USA, 3247-3252, 2009. http://doi.acm.org/10.1145/1520340.1520466
16. Gaskins N. R. ``Cognitive-Computer Artifacts for Urban Art and Communication" ACM, Austin, TX, USA, 2012