4 posts

Use Synapse to workaround AWS ELB static IP limitations


The system receives data from a third-party service using TCP sockets. The service requires a static IP address to send the data. Several app nodes are created to receive and process data. AWS ELB is used as the load balancer for app nodes. But currently AWS ELB only has host name, but no static IP address. AWS has elastic IP addresses which are static, but cannot be associated with ELB.

Solution #1 - HAProxy (Not working)

The first solution I had was to use a HAProxy server as the proxy to AWS ELB. Install HAProxy on one EC2 instance and assign an elastic IP address to it. HAProxy receives data and forward to ELB.

Issue with this solution is that HAProxy only resolves DNS names during start. So once HAProxy starts and IP address of ELB changes, there is no way to detect that and HAProxy keeps sending to old IP address.

Solution #2 - Synapse

Synapse is a service discovery system from Airbnb. It builds upon HAProxy. Synapse provides certain watchers which watch changes. Once changes are detected, Synapse generates a new HAProxy configuration and reloads HAProxy. Application talks to HAProxy instead of the actual proxied service.

Back to the problem, I used Synapse to replace ELB. Synapse has a watcher ec2tag which can watch tags of EC2 instances. To add/remove instances from Synapse, just add/remove certain tags. For example, Synapse watches tag name/value env=test of EC2 instances. Once a new instance with tag env=test is launched, Synapse detects this change and update HAProxy config file to include the new instance. The new instance now is able to receive data. Load balancing is provided by HAProxy.

Install Synapse

It's recommended to install Synapse directly from GitHub master branch. Release 0.11.1 has some issues. For example, if you're using bundler, add following to your Gemfile:

gem 'synapse', :git => 'git://'  

If you're using Chef, use gem_specific_install cookbook.

gem_specific_install "synapse" do  
  repository ""
  revision "master"
  action :install


Synapse configuration is done by a YAML file synapse.conf.yaml. In this file, you define services and HAProxy configuration.

          name: "elb"
          host: "<elb-host>"
          port: 7000
        method: "ec2tag"
        tag_name: "env"
        tag_value: "test"
        aws_access_key_id: "<aws-key>"
        aws_secret_access_key: "<aws-secret>"
        aws_region: "<aws-region>"
        port: 3200
        server_port_override: "7000"
        server_options: "check inter 2000 rise 3 fall 2"
          - "mode tcp"
          - "mode tcp"
    bind_address: ""
    reload_command: "service haproxy reload"
    config_file_path: "/etc/haproxy/haproxy.cfg"
    do_writes: true
    do_reloads: true
      - "log local0"
      - "log local1 notice"
      - "user haproxy"
      - "group haproxy"
      - "log global"
      - "balance roundrobin"
      - "timeout client 50s"
      - "timeout connect 5s"
      - "timeout server 50s"

In the Synapse config file above, services section defines different services to watch. For myservice, default_servers section contains the fallback servers when no servers are discovered, here I used the ELB server. discovery section contains configuration about different discovery methods. For ec2tag, you need to provide AWS access key & secret, region and tag name/value to watch. haproxy section contains local HAProxy configuration for this service. In the example, HAProxy for myservice listens on port 3200 and forwards traffic to app nodes no port 7000. The second haproxy section contains global HAProxy configurations.


Copy the YAML file to some place, e.g. /etc/synapse.conf.yaml, then start Synapse using synapse -c /etc/synapse.conf.yaml.

Solution #3 Nginx (untested)

Nginx seems to have better support of DNS resolution, so it may work to use Nginx as the proxy.

AWS Elastic Beanstalk - No carriage returns in Dockerfile

AWS Elastic Beanstalk has a very good Docker support. But for Dockerfile, you need to make sure no carriage return (CR) is in the Dockerfile, only line feed (LF) should be used. Otherwise, you may find weird issues.

For example, a simple Dockerfile as below,

FROM ubuntu:14.04  

If a CR is the last character in first line, Beanstalk will have error like repository not found ubuntu:14.04. This is because Beanstalk uses ubuntu:14.04<CR> as the repository name. So CR cannot be used in Dockerfile.

On Windows platform, Notepad++ can be used to find CR and remove them.

Install AWS CLI on Ubuntu

To install AWS CLI on Ubuntu, you need to use Python 2.7. If you use Python 2.6, there is a conflict error with simplejson library.

You may need to uninstall Python 2.6 first using sudo apt-get remove python. Then install Python 2.7.

sudo add-apt-repository ppa:fkrull/deadsnakes  
sudo apt-get update  
sudo apt-get install python2.7  

Then install pip using sudo apt-get install python pip, then install AWS CLI using sudo pip install awscli.