Road to startup infra - Part 1 - Certificate Authority

This is part I of an ambitious project: building a infrastructure for startup and the enterprise, create a kubernetes cluster, migrate workloads, design and deploy systems on top of it. All alone.

This post is a sensitive topic and I will probably not cover all the responsibilities on doing such thing.

This is a optional step in kubernetes cluster building, but it will pay off later.

I recommend that you plan in advice, and study how to keep the Root CA secure and offline for decades.

You could put your entire organization security on this point (a least a very sensitive one), is not a bad idea, and not as scary as it sounds, but strong security need to take that seriously.

The use case

We want the ability to emit your on certificates to secure communications between pretty much any part of your organization.

Some endpoints we will protect are:

  • Web servers

  • Kubernetes Helm

  • Kubernetes API

  • Any other server-to-server TLS, as well as server to employee communication.

The certificate will be valid in house, or any other entity that explicit trust your certification.

Create your own certificate is not recommended for consumer endpoints, because you need a certificate trusted by its devices. You could pay for it or use the awesome Let's Encrypt project. This is not a problem, and in many cases cheap, but by using your own CA internally you have these advantages:

  • Cost is free - You can issue any number of certificates you want.

  • You can automate this process

  • You can have ultimate control of it

  • If you distribute the certificate in a automated way (ansible, active directory), the use is transparent in the devices your organization controls.

As you probably have thought, the main disadvantages are:

  • Any device outside your control have to import the root and intermediate certificates and trust on it

  • The root certificate is the father of all, if you compromise it, you are condemned to death in hell. This is why Verisign, Geotrust and others must take first class very strong security measures to protect it, they companies will go out of business if a hacker gets it hands on it, and it will damage a uncountable number of clients and business across the internet. The reaction time (for any entity who trust the compromised certificate) is crucial in this case, but the damage will be remarkable.

The good news is that is not so hard to have a strong security on it. Unless the hacker have its on 500-2000 qubits quantum computer, them we can go home and stop using any electronics for transactions like your grandfathers live (I think I going to buy a piece of gold just in case). Intel announce a 49 qubits cpu (billions of dollars per chip), so we are progressing man :-). And by the way, a hacker with a quantum computer? No, but Bill Gates... Maybe... (its not about money but technology, this is interesting to search by the way)

What about external clients? I assume they will use web and https to reach your systems. So you can use Let's encrypt and Kubernetes Ingress to automate the process and get free of charge trustful SSL that is dead simple.

We going to talk about it in other posts. If you are curious check Voyager (Nginx Ingress has this feature as well)

The higher goal

Your goal can be divide in 5 simple steps:

  • Generate a root certificate, make security copy's of it.

  • Generate a Intermediate Certification Authority add it to Hashicorp Vault

  • Use vault to generate the certs

  • Deploy the public root and intermediate certificate on your machines

  • Use the certificates on your systems

Wait a minute, what is Vault? As the name says is like a Bank Vault, very secure (not as secure as a paper in a bank) designed to safe keep your certificates and electronic secrets and expose it for your systems.

I will not get in to details of it, check the project documentation. I assume you understand the basics of vault, are using a dev server to test, or have setup a production vault server. I will provide some instructions for the production vault server, in the way I did it, but off course I do not assume responsibility for anything, take that as a blueprint.

As a final note, comments and suggestions a very welcome :-)

Hands On

Talk is cheap, let’s get starting.

But first stop and encrypt your disk. To avoid side attack make sure you computer is complete encrypted (full disk encryption). You could use Bitlocker on Windows or VeraCrypt on both Windows and Linux or FileVault on Mac.

As always, the internet is full of smart people, we will build on top of this amazing shared knowledge.

Installing Production Grade Hashicorp Vault

This is relatively simple, download the binary and create a service and a configuration file for it.

    curl -O https://releases.hashicorp.com/vault/0.10.2/vault_0.10.2_linux_amd64.zip
    unzip vault_0.10.2_linux_amd64.zip
    mv vault /usr/local/bin/
    chmod +x /usr/local/bin/vault

Create the /etc/vault.conf configuration file:

    listener "tcp" {
        address = "0.0.0.0:8200"
        tls_cert_file="/etc/letsencrypt/live/vault.mycompany.com/fullchain.pem"
        tls_key_file="/etc/letsencrypt/live/vault.mycompany.com/privkey.pem"
    }

    backend "s3" {
        bucket = "company-vault"
        region = "eu-west-2"
        access_key = "xxxx"
        secret_key= "xxxxx"
    }

I will use AWS S3 to save the vault and Letsencrypt to provide SSL to it

S3 AMI policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::company-vault",
                "arn:aws:s3:::company-vault/*"
            ]
        }
    ]
}

As a harder security, make sure you do not have other policy with access to all your buckets. Vault is encrypted at rest, but you can still accidentally delete a file. I have not enabled versioning, but is probably a good idea to tweak that later (versioning and backups).

The systemd vault service:

[Unit]
Description=vault server
Requires=network-online.target
After=network-online.target

[Service]
PrivateDevices=yes
PrivateTmp=yes
ProtectSystem=full
ProtectHome=read-only
SecureBits=keep-caps
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.conf
KillSignal=SIGINT
TimeoutStopSec=30s
Restart=on-failure
StartLimitInterval=60s
StartLimitBurst=3

[Install]
WantedBy=multi-user.target

Install lets encrypt:

apt install letsencrypt

Get the certificate:

letsencrypt certonly --standalone -d vault.mycompany.com --test-cert

In the case you have nginx listen on port 80 (my case)

letsencrypt certonly --webroot --webroot-path -d vault.mycompany.com --test-cert

Remove the --test-cert when you are ready to get a real certificate

Nginx config (skip if you use standalone letsencrypt)

http {
    server {
        listen         80;
        location /.well-known/acme-challenge/ {
          root /var/www/letsencrypt/;
          default_type text/plain;
        }

        location / {
          return 301 https://$host$request_uri;
        }
    }
}

With the certificate on hands (it will land on /etc/letsencrypt/live/vault.mycompany.com) start vault server

    systemctl daemon-reload
    systemctl start vault

If is successful test the certificate:

openssl s_client -connect vault.mycompany.com:8200 | openssl x509 -noout -text

Certificate auto renew.

Letsencrypt certificates are short lived (3 months). Let's automate the process:

/etc/systemd/system/letsencrypt.service

[Unit]
Description=Let's Encrypt renewal service
Documentation=https://certbot.eff.org/#ubuntuxenial-nginx
After=network.target

[Service]
ExecStart=/usr/bin/letsencrypt renew --agree-tos
ExecStartPost=/bin/systemctl reload vault.service
Type=oneshot

/etc/systemd/system/letsencrypt.timer

[Unit]
Description=Daily renewal of Let's Encrypt's certificates

[Timer]
OnCalendar=daily
RandomizedDelaySec=10
Persistent=true

[Install]
WantedBy=timers.target
systemctl daemon-reload
systemctl start letsencrypt.service
systemctl start letsencrypt.timer
systemctl enable letsencrypt.service
systemctl enable letsencrypt.timer

Note: Weeks after this setup and after deploy of certificates to all servers and admin client machines, I remove the letsencrypt and nginx components and use the certificate generated by vault itself.

Good, Install vault on your admin machine and test the connection.

Keep the sealing keys secure. I'm not sure what is the most practical way to do it. There is 5 trust sources to unseal the vault. 5 people? When the service restart it will be sealed. My keys are just encrypted, when organization grows I will change and share keys. Manual process is not that good, but depends on reality, can you wait for manual unseal of vault? In a bigger company you will probably create a HA vault and route traffic only to unsealed instances, this is out of scope for this post.

Let’s move on.

Creating the Root and Intermediate CA

To avoid virus, leaks, and to copy/backup the keys in security, create a encrypted filesystem in VeraCrypt, and mount it. Put a strong password.

Yes, I'm telling you to full encrypt the disk, then create a encrypted mounted volume on top of it, this are two different protections:

  1. Full disk encryption will protect you against careless and side attacks (command history, memory pagination, ssh keys). But all files will be open for all programs once you boot
  2. The encrypted mounted filesystem will protect you against other process running on the PC, and from yourself (once sealed). And you can also makes copies of it, on other encrypted or non-encrypted filesystem's (accidental non encrypted backups for example). Just make sure you choose a strong password you will remember

Once is done. Follow this instructions jamielinux.com/docs/openssl-certificate-aut..

Create:

  • A root pair
  • One intermediate certificate for vault
  • One intermediate certificate for Kubernetes Helm (optional)
  • One intermediate certificate for each group of systems you want emit certification. Kind of logic isolation (optional)

Name convention: Company Ltd Root CA, Company Ltd Intermediate CA - Identification

Encryption: 4096bit

Hash: SHA-256

Validation: 20 years for the Root, 5 years for intermediates

Is not exaggerated to put a simpler different password on the keys, this is a third layer of security.

Make a copy of the public keys outside the mount

Do not seal the filesystem yet, we will upload the private key for vault ca. After this is done, seal the filesystem and make copies of it.

The intermediate certificates, I'm using for Vault CA and a CA for Kubernetes Helm. I think is a good idea to generate separate certification authorities per component, but the Kubernetes installation I use will generate certificates for components like kubeapi and etcd. If you are wondering what is helm, is a kubernetes package installer with two components: An client and a server. I will talk about it in the post Kubernetes Post Install

Generate two Intermediate CA: Company Name Ltd Intermediate CA - Vault, and Company Name Ltd Intermediate CA - Helm

Upload the certificates to Vault

First remove private key password:

openssl rsa -in private\helm.key.pem -out private\helm.key.nopass.pem

Then create a bundle with pem and key:

cat certs\helm.cert.pem private\helm.key.nopass.pem > private\helm-pembundle.cert.pem

Prepare vault and upload. In this example I will upload the helm intermediate, we mount a pki for each intermediate. The Vault Intermadiate will be on default pki.

  1. Mount a new pki: vault secrets enable -path=pki_helm pki
  2. Check the secrets: vault secrets list
  3. Config the certificates: vault write pki_helm/config/urls issuing_certificates="https://vault.example.com:8200/v1/pki/ca" crl_distribution_points="https://vault.example.com:8200/v1/pki/crl"
  4. Upload the cert: vault write pki_helm/config/ca pem_bundle=@private\helm-pembundle.cert.pem

Remove the no passwd key and pembundle

rm private\helm.key.nopass.pem private\helm-pembundle.cert.pem

Note: You can create a certificate, with sign request in vault, and never have to leave the private key outside hashicorp vault. However, I do not know how to do it appropriated for a root ca. It is easier to generate with openssl and upload.

Now you can seal the filesystem

Using vault to generate certs

First, is good to tweak the max validation for all certification emitted by vault:

vault secrets tune -max-lease-ttl=8760h pki

Second, update the CRL location and issuing. Revocation is a whole beast, but start with this, better than nothing I think. Good read on the topic: maikel.pro/blog/current-state-certificate-r..

vault write pki/config/urls \
    issuing_certificates="http://127.0.0.1:8200/v1/pki/ca" \
    crl_distribution_points="http://127.0.0.1:8200/v1/pki/crl"

Of course, change the url to your vault dns address.

Now create a role to emit certificates

vault write pki/roles/my-role \
    allowed_domains=my-website.com \
    allow_subdomains=true \
    max_ttl=8760h

Generate the cert

vault write pki/issue/my-role \
    common_name=service.my-website.com

Deploy the public root and intermediate certificate on your machines

This is easy. On Ubuntu servers and Desktops add the certificate on /usr/local/share/ca-certificates and run update-ca-certificates command.

You could use this Salt Stack pillar to deploy on all servers. Just put your public root certificate on the folder.

Note: You should put only the root certificate on the truststore of the machines. That way the chain will be trusted. If you put the intermediate as well, it will work, but the chain will not show.

    {% if grains['os_family'] == 'Debian' %}
    deploy ca files:
    file.recurse:
        - source: salt://common/files/ca
        - name: /usr/local/share/ca-certificates
        - user: root
        - group: root

    'update-ca-certificates':
    cmd.run:
        - onchanges:
        - deploy ca files
    {% endif %}

Ansible is a good choice as well.

For windows clients, you could follow this instructions thewindowsclub.com/manage-trusted-root-cert..

Use the certificates on your systems

Well you know how to do it ;-)

Check and show the certificate:

openssl s_client -connect yourserver.company.com:443 | openssl x509 -noout -text

List of useful commands:

gist.github.com/webtobesocial/5313b0d7abc25..

sslshopper.com/article-most-common-openssl-..

Conclusion

Now we have a fully function Certificate Authority. If you follow me on this:

  • We did install Hashicorp Vault
  • We did generate a secure off-line root ca.
  • We did create intermediated certificates to work on it and upload to vault.
  • We did use vault to generate certification and to automate the process. I let the automation for you to implement, but even without, generate a certificate is one command away.
  • We did trust the root on all server machines. Again the implementation was left to you, piece of cake using infrastructure automation.
  • We did follow paranoid strong security measures. The kind of Edward Snowden could be proud, not sure about bitlocker (Microsoft can send a copy key to its servers without you knowing).

About security: Security is much more than what we did. I made a blog about my recent paranoid actions, I predict that at some point NSA will test me to join as an agent :-). Joking aside, check this post

Next posts we will keep going:

  • We will automate infrastructure provision. In other words infrastructure as a code.
  • We will install a High Available Kubernetes Cluster.
  • We will install helm package manager on Kubernetes and use your certificates.
  • We will install ingress and external DNS and use lets encrypt with Kubernetes.
  • We will deploy your first service on Kubernetes and show it for the world.
  • We will discuss this infrastructure advantages and limitations.
  • We will have two infrastructures: VM's and containers. I will show what the state of it is and how I managed to maintain it.
  • We will backup everything and have a functional disaster plan.
  • We will monitor all the things.
  • We will design systems for scale and take advantage of kubernetes.
  • We going to migrate workloads for Kubernetes on the long term.

That is a lot work.