Kubernetes Proxy: Envoy vs NGINX vs HA Proxy

Having spent quite some time with Linux and Kubernetes admins, I've come to realize that networking isn't one of their strong sides. Being a network guy myself, I feel obliged to share my views on topics as important as this one. So, which proxy should you use in your Kubernetes cluster?

Lets start with some facts:
  • All three of these proxies are highly reliable, L7, proven proxies, with Envoy being the newest kid on the block.
  • All these proxies do an outstanding job of routing traffic L7 reliably and efficiently, with a minimum of fuss.
  • There is no full parity of features, but you can implement any critical missing features in the proxy itself… the power of open source!

To keep the post structure, just a few lines about each of these 3 Proxies:
  • HA Proxy is the default Load Balancer when it comes to Kubernetes. It was initially released in 2006, when the Internet operated very differently than today, ergo… there's an issue of slow adoption of new features. This is very serious when you consider SECURITY, like support for last SSL/TLS versions.
  • NGINX is a high-performance web server, FASTER and more modern then HA Proxy Load Balancer, WAF and so many other things… and if you check out the SDN integrations (Cisco ACI, VMware NSX, Nokia Nuage), these are all based on open source version of NGINX. NGINX open source has a number of limitations, including limited observability and health checks, so it comes down to what you're looking for. If you want an enterprise product, depending on your company environment - go with NGINX Plus, ACI or NSX (be sure to ask for -T).
  • Envoy Proxy is new… so not very mature, BUT - most modern, and used in production in Apple, Google among others. Envoy was designed from the ground up for microservices, with features such as hitless reloads, resilience, and advanced load balancing, plus - and exposing dynamic APIs for configuration. THIS is a big deal, in the world where proxies have been configured using static configuration files (Envoy also supports static config, of course). And lets not forget that Istio Service Mesh, which I'm a big fan and contributor of, uses an extended version of the Envoy proxy.

How I prepared for AWS SA Professional exam

Last week I managed to pass the AWS Solution Architect professional certification exam. Here's my certification, in all its glory:

If you've been following my blog, you'll know that I passed a Google Cloud Professional Architect exam in March. I wrote a few blog posts about how I prepared it, and you may find it all here.

Even though I've been preparing for the AWS exam for quite a while, the two main reasons I went for GCP professional level exam first are simple:

  • I think Google Cloud is a sleeping giant, and I wanted to be among the first certified experts. 
  • AWS has much more services. For a professional level exam you don't just need to know some of them in depth, you need to know ALL of them in depth, in order to make the right architecture that fits the customers requirements.

How I prepared


  • Linux Academy has amazing hands-on courses for both Associate and Professional level. In my experience - the only one that really prepare you for this exam.
  • Work experience. This is where it gets tricky… AWS has a wide service catalogue, and your work environment hands is unlikely to cover the entire blueprint.

Difference between AWS Associate and Professional level exams

This is something I get asked a lot. Here is the main difference:

  • To pass the associate level exam, you need to know what each service does. The questions are straight forward, if you know what the service does - you'll eliminate most of the options in your test, and get the right answer.
  • AWS SAP (Solutions Architect Professional) is a real world business problem oriented exam. It's understood that you know all the AWS Service Catalogue in depth, and you are tasked with getting the most optimal architecture based on the customer requirements. You will get 77 different business scenarios (this is a LOT of text, so be prepared), and each one has 4-5 possible answers, which are all correct, you just need to figure out which one is the best for that particular scenario.

This basically means that if the question is how to connect your VPN with your on-premises infrastructure in the most cost efficient way, the answer will vary:

  • In Associate level, you will go with VPN IPSec, cause Direct Connect is more expensive.
  • In Professional level you'll have to go deeper, and it's likely that mapping the use case with the architecture, Direct Connect could come out as the most cost efficient option.

AWS vs GCP professional certifications

This is a tricky one… Basically this is how it is:

  • GCP exam is very, very difficult. I feel like it's a Cloud Architect and DevOps merged into one exam, which makes it quite complex and "uncomfortable" at moments. BUT - GCP doesn’t have nearly as many services as AWS does in the Service Catalogue, so I guess the blueprint is narrower, which kind of justifies the complexity of the exam.
  • AWS is difficult, and long, requires high concentration during the 170 minutes, and probably what I like more - tests you for the real world skills. You will potentially get the same possible architectures as the answers in many different questions, and I feel it's impossible for someone to pass it even if they knew the questions, you really need an architect mind. On the positive side - there are no trick questions, so if you're good - you'll pass, it's as simple as that.

What's next? 

I'm going all in for my VMware VCDX (Design Expert) exam now. Did the design, going for the defence. I think I'm in the point in my career to go for something like this, get roasted for thinking I'm a super architect… Bring it on, my ego is about to be destroyed, but I feel like I'll come out of the experience as a true business architect.

On relevance of CCIE in 2019

A question I've been getting a lot from the Network Engineers, should they go for CCIE. There are two points to this question:

  • Knowledge and skill
  • Value of CCIE as a Certification

Let me get into more detail.

Value of CCIE as gaining skill and knowledge

Networking as such is changing. A network engineer for the cloud era needs to understand programmability, APIs, SDN with its use cases, Public Cloud networking (inter and intra public cloud). BUT, if you've ever talked to a network engineer who doesn't come from hardcore cisco or juniper networking, and rather comes from systems (VMware or Linux), or someone who's just studied something like OpenFlow and considers hardware to be a "commodity", you'll notice how due to lack of basic networking L1-4 concepts, they tend to not understand some limitations in both functionality and performance. There are exceptions, of course, and I want to acknowledge that!!! The point I'm trying to make is that CCIE gives you the best of breed base for any kind of programmable, cloud, Kubernetes or whichever networking-related activity you want to pursue in the future.

Value of CCIE as a Certification

This is a completely different topic. If you want to do your CCIE just because you want more money from your employer - don’t. Go learn AWS, learn Python and Ansible, maybe some ACI and NSX but from the "north side" (API). The days when getting a CCIE meant an immediate salary increase of 50% are over… It is now a step in your trip, not the final goal.


Should you go for a CCIE? Yes. If you are serious about networking, you 100% should. You will learn all that other SDx and Cloud stuff much more easy if you understand bits and bytes. Hey, I passed my Google Cloud, AWS, and NSX highest level technical certifications greatly thanking to the networking knowledge I learned working on the field as a CCIE... I'm just doing Networking in a different way now. But - it's still networking, L2 and L3, same old MAC, IP and BGP, just consumed in a different way.

Just married: IBM and RedHat. What does this mean for Cisco and VMware Multi-cloud offer?

As per yesterdays announcement, IBM is acquiring Red Hat in deal valued at $34 billion (more about this here). This is another one in a row of deals I did not expect to happen:

  • Oracle acquired Sun Microsystems
  • Microsoft acquired GitHub
  • Dell acquired VMware

How disruptive can a Purple Hat really be? VMware survived being acquired by Dell quite well... will RedHat have the same luck, or not? What I know for sure is that the RedHat employees are panicking right now...

Sure, 3k billion is a big sum, but also a bold move by IBM on the conquest to the Multi-Cloud market. Combined we're looking at (to name a few):

  • Ansible for the Automation
  • OpenShift, as the best of breed PaaS based on Kubernetes
  • CloudForms as a potential CMP (I wonder how this will work out...)
  • Watson for all AI/Machine Learning related
  • IBM Cloud as a Public Cloud platform

Is this a winner combo? Or do other Hybrid Cloud promoters, like Cisco and VMware have equally good lock-in-free proposals?

As a Hybrid Cloud and DevOps advocate, and a European CTO, I've had the experience to "casually chat" to many European companies about their Cloud strategy. Two things are evident:

  • The buyer is changing, Multi-Cloud is an APPLICATION strategy, not the infrastructure strategy (read more about this here).
  • Companies don't really know who to trust, as what they're being told by various vendors and providers is not really coherent. This makes is pretty difficult to actually build a Cloud strategy (don't get me started on CEOs who'll just tell you "We've adopted Cloud First", and actually think they have a cloud strategy).

Due to all this:
- IBM and RedHat, as software companies, will be able to get to the Application market.
- Neither of the two can do Infrastructure as well as VMware & Cisco.

How important is this? Very! And here is why.

Cisco has:

  • Cloud Center, a true application oriented micro-services ready CMP, Public Cloud and Automation Tool agnostic, equipped with the right Benchmarking and Brokering tools, that integrates quite well with the infrastructure, and workflow visibility platforms.
  • ACI and Tetration, that enable the implementation of coherent and consistent Network and Security Policy Model across multiple private and public clouds, along with the workload visibility.
  • HyperFlex and CCP, providing enterprise production-ready, lock-in-free Kubernetes solution on a Hyper Converged infrastructure.
  • AppDynamics and Turbonomic, a true DevOps combo for the Day 2 we're all fearing in the Cloud, letting the application architects model their post-installation architecture, and monitor the performance of each element, latency between different elements, and assure the optimal user experience.

VMware has:

  • vRealize Automation, the best of breed Automation and Orchestration Hybrid Cloud ready platform.
  • PKS and VKE, KaaS platforms that provide the enterprise production-ready Kubernetes solution, with a fully prepared Operations component, in both - private and public cloud.
  • Wavefront, application visibility tool running on Containers, designed with Cloud applications and Micro Services in mind, with just insane performance.
  • NSX, including the full SDN stack in both, Data Center and Cloud, with probably the best API (both documentation and usage wise).
  • Partnership with AWS, Azure, GCP and IBM, to leverage the most demanded Hybrid Cloud use cases in a "validated design" fashion.

What does this all mean?

Multi-cloud is still a space that, based on Gartner and IDC, over 90% of Companies are looking at. Big companies are making their moves... so just grab your popcorn, and observe. It's going to be a fun ride!

Why SDN isn't where we thought it would be

The SDN hype started a few years ago. Everyone was talking about it as the next big thing, and it all made so much sense. I started exploring SDN while Nicira and Insieme were just two startups, and got even deeper into it when they were bought by VMware and Cisco and rose as ACI and NSX.

SDN makes perfect sense. A single point of management and operations of the entire data center network, micro Segmentation as an embedded feature, REST API support for automation, possibility to move the workloads between Sites without having to reconfigure the security policy, and a bunch of others. It truly is a missing piece, arriving a bit too late. So… why hasn't the same happened like when we started using server virtualization? Why isn't everyone implementing these technologies, and celebrating the benefits while singing their favourite tune?

In my opinion, two reasons: misleading PowerPoints and vendors with the wrong go to market strategy.

Misleading PowerPoints

Networking tends to be more complex then Compute and Storage in the Data Center. You have a group of independent network devices that need to transfer an insane number of packets between different points, with zero latency, and no time to talk to each other and coordinate the decisions. When you introduce Automation into the equation, it all gets really interesting. With SDN we introduced overlay, and managed to somehow make all this easier. Where is the problem then?

Automation is an awesome concept. If you automate, you will improve the delivery times, and always end up with the same results. Automation is not new… it's been here since the 70s, and even though the execution premises have changed, one thing stays the same:
- If you automate, you will save a lot of time and resources.
- To create automation, you need a lot of knowledge, experience and a lot of time and effort.

The truth about misleading PowerPoints lies in the second point. Everyone rushed to explain to their customers how their SDN has an API, and how you can automate everything in an instance, I saw bonus hungry AMs and SEs singing the songs to the customers about how they can use the automation tool of choice. "There's an API so you're good bro!" Unfortunately, this is far from the truth. Yes, SDN supports automation of your network, but it takes a lot of hard work to set it up right, and if you sold something to the customer without setting their expectations right… well, he will be disappointed.

What is the truth? Both ACI and NSX are mature solutions, but the SDN is no longer a group of independent switches, it needs to be integrated in the wider ecosystem, and it makes all the difference who integrates all this in your Data Center. If the customers were prepared for this from the beginning, I think we'd all bee seeing a whole lot more SDNs.

Vendor Strategy

I'll talk about 2 big ones here - VMware and Cisco. Have you noticed how these two vendors have the same number of production references in each moment? Like there is some kind of secret synchronisation behind the curtain. Ever wondered why that is?

The truth is that both, ACI and NSX, are great products. Yes, GREAT! It's also true that a surprisingly small number of "SDN experts" out there understands HOW and WHY these products need to be introduced in the data center ecosystem, so a majority of the happy SDN customers that Cisco and VMware are referencing are kind of fake, meaning - yes, they are using the product, and yes, it's in production, but it is not used as SDN. Sure, Cisco has DevNet, and VMware has VMware Code, and these are both great initiatives, but they still lack a critical mass… both of them do. [if you don’t know what these are, I STRONGLY recommend that you stop reading this post, and go check out both these websites, they are AWESOME].

What is Cisco's mistake?

Cisco counts on their traditional big partners to deliver ACI. These guys can sell networking to a networking department, they get BGP and VxLAN, they can build the fabric in what Cisco brutally named "a networking mode", and they can train the networking department to use ACI. That’s it. So… what about automation, IAC (infrastructure as code), what about the developers who are actually the true buyer here, and they just need to provision some secure communication for their code? Well, I'm afraid there's nothing here for them, because neither a Cisco's networking partner not the customers networking department are able to configure and prepare the ACI for what these guys really need SDN for. Customer simply isn't getting what they paid for, and they are pretty vocal about it in the social networks, so the product gets bad marketing.

And yes, there are companies out there (such as mine) who are able to implement ACI as a part of a Software Defined ecosystem and help customer build automation around it, but Cisco somehow still isn's seeing the difference, and is still promoting same old networking partners to the customers to implement their ACI. Oh well… let's hope Cisco starts to understanding this before it's late.

What is VMware's mistake?

NSX is an entirely different story. The problem isn't VMware's strategy, but rather - the buyer. SDN is still networking, so the buyer is a Networking Department, but… Networking guys don’t know VMware, they know Cisco and Juniper. On the other hand there are System Admins who are desperate to gain control over network and not depend on the slow networking departments, but… they lack advanced networking knowledge. So NSX, being a brilliant product as it is, ended up in no mans land. VMware did everything to promote NSX to Network experts, if you're a CCIE, like me, you can actually do NSX cert exams without doing the training, and NSX is easy to learn and understand, but still, not enough hype around it among network admins. So, what happened? Well, for now there are many implementations of NSX used the way System and Security experts are able to promote and manage it, Micro Segmentation with some basic networking, but not even close to the NSX full potential, and again - not used as an SDN.

What about the other SDN vendors? 

There are a few worth mentioning: Nokia Nuage, Juniper Contrail, some distributions of OpenDayLight (HP, Dell, Huawei, Ericsson, NEC, etc.). Two things are happening with these guys:

  • Due to all mentioned above, the Customers are under the wrong impression that not even ACI and NSX are fully mature and stable solutions… If Cisco and VMware aren't able to invest what it takes and make it stable, what do you expect from the others?
  • In one moment all these guys made huge investments in their technology, and there was still no sales to support the investment, so - they lowered the prices and started selling the solutions that weren't yet mature. This caused customers dissatisfaction, and the rumor on the market that SDN just "isn't there yet". They can still recover… as long as they actually invest in product development and engineering skills, and let product sell itself. 

What should we expect in the next 2-4 years?

SDN is here to stay, even more so with IoT and Containers with a whole set of new micro Segmentation and Network automation requirements. It just takes it longer then anticipated to find it's place. I think the customers are slowly starting to get the non-planned effort to actually move from installing the SDN product - to using it as a Software Defined technology, which is good, so if you're considering SDN as a potential career path - add some automation and programming skills, and you're on the right track.

Migrate HyperFlex Cluster to a new vCenter

Get ready to have your mind blown. One of the easiest procedures I've encountered. You just need to follow these 3 steps, to migrate the entire HyperFlex vSphere Cluster with all its hosts from vCenter 1 to vCenter 2.

Before you start:

- Your environment might be different. I'm not responsible if something goes wrong, you're welcome to look for the official guides. I've tested it to migrate from vCenter 6.0 to 6.7 in August 2018.
- VDS WILL NOT be migrated automatically, BUT - you can Export it into ZIP from the old vCenter, and import into the new one AFTER you've done all these steps, and the Uplinks will be automatically mapped. Be sure to include all the configuration, portgroups and all, both when you export and import.

Step 1.

Deploy your vCenter Server Appliance. I'll asume you're setting the standard username, administrator@vsphere.local

Step 2.

Create both Datacenter and Cluster in the empty new vCenter. For the ease of migration, use the same names. Connect all ESXi hosts from HyperFlex to the new Cluster. Just accept re-assigning of the licence, and wait to see the new host as Connected.

Step 3. 

Re-register the Cluster to a new vCenter. I recommend that you observe the new vCenter in the background, so that you can follow the progress. To do this you need to SSD into your HyperFlex, and execute the following command (set your own parameters, of course):

stcli cluster reregister --vcenter-cluster CLUSTER_NAME --vcenter-datacenter DATACENTER_NAME --vcenter-password 'NEW_vCENTER_PASSWORD' --vcenter-url NEW_vCENTER_IP --vcenter-user administrator@vsphere.local

You will get this message:

Reregister StorFS cluster with a new vCenter ... [this is where you wait for approx 10 minutes]
Cluster reregistration with new vCenter succeeded

Additional Step:

If you are using VDS, this is when you need to import them to the new vCenter.

And - you're done! Let me know in the comments if it worked as easy as this.

Install PowerCLI on Mac, start using PowerNSX

This is something I've been wanting to publish for a while, and finally my Mac got formatted (no questions will be taken at this point...) and I had to re-install it all, and I just couldn't find the instructions on how to do it without just having to read pages and pages of disclaimers and stuff...

Why PowerCLI? Cause it's a simplest way to automate your vCenter tasks, via the command line, fast and furious. Sure, one day a working vCenter web plugin will come, but who knows when...

Why PowerNSX? Same... but for the NSX admins. Trust me, my life got so much better the day I stopped depending on vCenter Web GUI.

How do I install and start using it? Simple. Just follow this 5 Steps guide...

Step 1: Install PowerShell (check the update below first!!!)

Make sure you have GitHub:

# git

Clone the PowerShell installation package from GitHub:
# git clone --recursive https://github.com/PowerShell/PowerShell

Once you got it, enter the Folder, and install the package (you'll be asked for a Password a few times):
Submodule path 'src/libpsl-native/test/googletest': checked out 'c99458533a9b4c743ed51537e25989ea55944908'

MatBook-Pro:~ mjovanovic$ cd /Users/mjovanovic/PowerShell
MatBook-Pro:PowerShell mjovanovic$ ./tools/install-powershell.sh

Get-PowerShell Core MASTER Installer Version 1.1.1
Installs PowerShell Core and Optional The Development Environment

Run "pwsh" to start a PowerShell session.
*** NOTE: Run your regular package manager update cycle to update PowerShell Core
*** Install Complete

MatBook-Pro:PowerShell mjovanovic$ pwsh
PowerShell v6.0.2
Copyright (c) Microsoft Corporation. All rights reserved.

Type 'help' to get help.

PS /Users/mjovanovic/PowerShell>

You're in the PowerShell!!!

UPDATE: As of December 2018, this method is no longer supported. You'd actually get into a quite "nifty" catch 22, where PowerShell 6.1.1 doesn't support most of relevant PowerCLI Commands, New PowerCLI doesn't support anything under 6.0.5, and some PowerNSX Commands require 6.0.1 and above. Awesome!

SOLUTIONDownload PowerShell 6.0.5, that one works! Download it as a package, and install. The rest of the post remains the same.

Step 2: Install PowerCLI

Now lets procede with the PowerCLI. More details on this Link, if you happen to need more details... but basically all you need is the following command: https://blogs.vmware.com/PowerCLI/2018/03/installing-powercli-10-0-0-macos.html

PS /Users/mjovanovic/PowerShell> Install-Module -Name VMware.PowerCLI -Scope CurrentUser

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install
the modules from 'PSGallery'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): Y

Step 3: Install PowerNSX Modules

Ok, so now we just need to install the PowerNSX Modules:

PS /Users/mjovanovic/PowerShell> Find-Module PowerNSX | Install-Module -scope CurrentUser

Untrusted repository
You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install
the modules from 'https://www.powershellgallery.com/api/v2/'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "N"): Y

Step 3.1: Resolve the Certificate Error:
If you tried to connect to your vCenter now, you´d get this error:
Connect-VIServer : 06/08/2018 18:32:43 Connect-VIServer Error: Invalid server certificate. Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Ignore to ignore the certificate errors for this connection.

Before Logging in to your vCenter, to avoid the Certificate problems (which you will most definitely have), first use, You need to set the Certificate Errors to FALSE:

PS /Users/mjovanovic/PowerShell> set-PowerCLIConfiguration -InvalidCertificateAction Ignore

Perform operation?
Performing operation 'Update PowerCLI configuration.'?
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [S] Suspend  [?] Help (default is "Y"): Y

PS /Users/mjovanovic/PowerShell>

Step 4: Log into the NSX Manager and vCenter

Now you are GOOD TO DO, you can Log in to your NSX, and to the vCenter:

PS /Users/mjovanovic/PowerShell> Connect-NsxServer -NsxServer -Username admin -Password M4TSCL0UD

PowerNSX requires a PowerCLI connection to the vCenter server NSX is registered against for proper operation.
Automatically create PowerCLI connection to
[Y] Yes  [N] No  [?] Help (default is "Y"): Y

WARNING: Enter credentials for vCenter

PowerShell credential request
Enter your credentials.
User: administrator@vsphere.local
Password for user administrator@vsphere.local: **************

Version             : 6.4.0
BuildNumber         : 7564187
Credential          : System.Management.Automation.PSCredential
Server              :
Port                : 443
Protocol            : https
UriPrefix           :
ValidateCertificate : False
VIConnection        :
DebugLogging        : False
DebugLogfile        : \PowerNSXLog-admin@

Step 5: Start using PowerNSX

You can do so many things here! I recommend this Guide to get you started:

Most important command:
PS /Users/mjovanovic> get-command -module PowerNSX                                                                                                                                 

CommandType     Name                                               Version    Source                                                                                              
-----------     ----                                               -------    ------                                                                                              
Function        Add-NsxDynamicCriteria                             3.0.1118   PowerNSX                                                                                            
Function        Add-NsxDynamicMemberSet                            3.0.1118   PowerNSX                                                                                            
Function        Add-NsxEdgeInterfaceAddress                        3.0.1118   PowerNSX                                                                                            
Function        Add-NsxFirewallExclusionListMember                 3.0.1118   PowerNSX                                                                                            
Function        Add-NsxFirewallRuleMember                          3.0.1118   PowerNSX                                                                                            
Function        Add-NsxIpSetMember                                 3.0.1118   PowerNSX                                                                                            
Function        Add-NsxLicense                                     3.0.1118   PowerNSX                                                                                            
Function        Add-NsxLoadBalancerPoolMember                      3.0.1118   PowerNSX                                                                                            
Function        Add-NsxLoadBalancerVip                             3.0.1118   PowerNSX                                                                                            

Function        Add-NsxSecondaryManager                            3.0.1118   PowerNSX                     

Just play around with these!

Most Popular Posts