AWS Lambda has been HIPAA eligible for a month, and it's awesome

What is Lambda and why you should care

AWS Lambda is the standard bearer of a new paradigm in cloud computing generally referred to as "serverless". Before serverless, developers needed to provision servers in the cloud. They needed things like patches, health checks, monitoring software, and all of that takes a toll on the people responsible for keeping them running.

The serverless architecture removes almost all of those concerns, and lets you simply drop code into the cloud and have it run in response to pre-defined events. The cost structure is totally different—rather than paying for a server that is always running, you just pay for each invocation of the function.

Why healthcare can benefit from Lambda

In October 2017, AWS announced a huge swath of services becoming hipaa-eligible. Among them was Lambda.

Amazon's announcement immediately sparked excitement on our Slack channel, as we have been using Lambda since 2015 for a host of tasks ranging from closing old Github issues and notifying our customer contacts when problems arise to tidying up our production infrastructure. Up until October 2017, we were extremely clear about the boundaries of what could go in Lambda and all of our clinical data processing pipeline was on EC2, no exceptions.

Redox had immediate use cases for a nimble, on-demand way to process data. Chief among which was processing data coming into to our SFTP server.

A surprisingly large amount of healthcare data exchange happens over SFTP. When it does happen, there's also a lot of custom processing that goes along with it. We were staring down a project that would involve unzipping a file from the SFTP server, taking the resulting delimited files and stitching them together, and then loading into the Redox Engine for routing and further processing.

How we ended up using Lambda

My outstanding colleague Dani Perszyk did all of the design and development on this project. She came up with some really cool techniques that we're going to continue to use in the future.

Use a Serverless framework - We arrived on the Serverless framework to help with a lot of the heavy lifting. Other teams have used Apex in the past, and both are great. The bottom line is that deploying your Serverless code should be as easy as possible. With Serverless, it's as easy as one command in our CI file and updates are live.

Understand the events that can trigger a function - As I mentioned, this project required custom processing after receiving a file over SFTP. We built the SFTP server to only interface with S3—it literally never touches the file system, so we were already in great shape. If you're not already using AWS services such as S3, SQS, or API Gateway, the lift in deploying new Lambdas could be slightly higher.

Understand the AWS-imposed limitations - You should come in knowing that there are real memory limits and time limits on how long a Lambda function can run. Design your processing to be concurrent, or at least be able to stop and resume.

You're pretty much on your own for monitoring - A huge part of the service we provide is monitoring. It's one thing to be able to write a custom file parser, it's way harder to do it reliably and fix issues when they come up. Serverless has a few plugins that can help with getting logs to AWS cloud watch and similar, but we needed to build a PagerDuty API integration directly into the app.

Conclusions

If you're not already familiar with Lambda, or similar services like Google Cloud Functions and Azure Functions, you're missing out on a technology that can save you money and simplify your developer flow.

As of now, only AWS Lambda and Azure Functions are hipaa-eligible, so do your due diligence when deciding to process protected health information (PHI) in a serverless architecture.

If you're a company like Redox that does a lot of batch processing of data, or even a company that consumes data from the Redox API, you could potentially save a lot of maintenance headache and save a lot of money by moving to a serverless architecture. Again, do your due diligence—we're only using Lamda to process files from our SFTP.