tag:blogger.com,1999:blog-60916451171725615422024-03-17T09:04:41.679+01:00Welcome to Mat's CloudHitchhikers Guide to Hybrid Cloud, by Mat Jovanovic (@matjovanovic)Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.comBlogger148125tag:blogger.com,1999:blog-6091645117172561542.post-27235856476807076672020-04-18T00:10:00.001+02:002020-04-18T08:41:56.803+02:00AWS CDK: How to use CfnResource or Cfn Layer<h3>
What is AWS Cloud Development Kit?</h3>
<br />
If you've been in the world of AWS long enough, you've probably heard of CloudFormation (duh!) and AWS CDK (Cloud Development Kit). If you haven't - you should! Trust me, it's awesome!!! AWS CDK gives you the best of Imperative and Declarative when it comes to IAC (Infrastructure as Code). It allows you to:<br />
<br />
<ul>
<li>Keep the entire code of your app on Git as a "single source of truth", making the best use of SCM tools such as version control and CI.</li>
<li>Use your favourite programming language (read: python!... and yes, I know AWS recommends TypeScript and the documentation is much better, but I'd still go with Python). In the end of the day, CDK is just a way for you to create and deploy CloudFormation Stacks.</li>
</ul>
<div>
<br /></div>
<div>
<h3>
How do I get started with CDK?</h3>
I'd highly recommend you to get familiar with CDK, and I'll leave 3 links you need to bookmark:</div>
<div>
<ul>
<li><b>Official AWS documentation</b>: <a href="https://docs.aws.amazon.com/cdk/latest/guide/work-with-cdk-python.html">https://docs.aws.amazon.com/cdk/latest/guide/work-with-cdk-python.html</a></li>
<li>An awesome <b>CDK Workshop</b> that you MUST do: <a href="https://cdkworkshop.com/30-python.html">https://cdkworkshop.com/</a> . You'll see that most of blogs and youtube videos called "Intro to CDK", are simply a rip off of this workshop.</li>
<li>Official AWS CDK <b>API Reference</b>: <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/aws-construct-library.html">https://docs.aws.amazon.com/cdk/api/latest/docs/aws-construct-library.html</a></li>
</ul>
<div>
<br /></div>
</div>
<div>
<b>Piece of advice</b>: use all these docs to create your own "how to", because... you'll see, the docu<span style="font-family: "times" , "times new roman" , serif;">mentation still needs some work.</span></div>
<div>
<span style="font-family: "times" , "times new roman" , serif;"><br /></span>
<br />
<h3>
<span style="font-family: "times" , "times new roman" , serif;">
CDK Building Blocks</span></h3>
<div style="font-size: 12pt; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "times" , "times new roman" , serif;"><b>Constructs</b> are the basic building block of CDK apps. They represent abstract “cloud components” which can be composed together into higher level abstractions via scopes. Scopes can include constructs, which in turn can include other constructs, etc. Constructors must always have:</span><br />
<span style="font-family: "times" , "times new roman" , serif;"> - <b>Scope</b>, which is normally 'self'</span><br />
<span style="font-family: "times" , "times new roman" , serif;"> - <b>Id</b>, as a local identity</span><br />
<span style="font-family: "times" , "times new roman" , serif;"> - <b>Kwargs</b>, a set of optional initialisation arguments, specific to each construct. For example, for Lambda, it will be runtime, code and handler, <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/aws-lambda-readme.html" target="_blank">more details here.</a></span></div>
<div style="font-size: 12pt; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "times" , "times new roman" , serif;"><br /></span></div>
<div style="font-size: 12pt; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "times" , "times new roman" , serif;">The AWS CDK is shipped with an extensive library of constructs called the <b>AWS Construct Library</b>. The construct library is divided into modules, one for each AWS service. For example, if you want to define an AWS Lambda function, we will need to use the AWS Lambda <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/aws-construct-library.html" target="_blank">construct library.</a></span><br />
<br /></div>
</div>
<h3>
<span style="font-family: "times" , "times new roman" , serif;">
How complex is CDK?</span></h3>
<div>
It's actually pretty easy, if you're using the officially supported classes. But... what if you need "more stuff"?</div>
<div>
<br /></div>
<div>
Let me clarify this:</div>
<div>
<ul>
<li>AWS CDK is awesome, but it can be challenging to use, since the documentation is not the best... especially if you're not using TypeScript.</li>
<li>In the official API reference, you'll see that there's an official way to create any resource. For example - for a DynamoDB Table, you'd go with an official class, MyDynamoDB = ddb.Table(), documented <a href="https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_dynamodb/Table.html" target="_blank">here.</a></li>
<li>Problem? As you can see in the link above, not all parameters are supported (panic mode: ON).</li>
<li>Solution: EVERYTHING you can do in CloudFormation, you can do using CDK (panic mode: OFF). You just need to not be scared to get electrocuted... a little. Every Official Class, has an CFN equivalent, that lets you use the EXACT parameters you're using to create CloudFormation Stack, but... in a different way.</li>
</ul>
</div>
<div>
<br /></div>
<h3>
Syntax for S3 CfnBucket or DynamoDB CfnTable</h3>
<div>
The whole reason I'm writing this, is that the official documentation, <a href="https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html">https://docs.aws.amazon.com/cdk/latest/guide/cfn_layer.html</a> is simply wrong. </div>
<div>
<br /></div>
<div>
We basically need to convert our good old CloudFormation Template written in JSON or YAML, into a Python code. Problem? Syntax isn't documented anywhere, and there are no examples on the web. I've spent the last few days trying to google "DynamoDB CfnTable KeySchema"... 0 hits! Even Google doesn't know about this.</div>
<div>
<br /></div>
<div>
Let's cut to the chase, let's say we've got the following CF Template in YAML, and we need to create a CDK Python code:</div>
<div>
<br /></div>
<br />
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #569cd6;">Resources</span><span lang="en-US" style="color: #686868;">:</span></div>
<blockquote class="tr_bq">
<!--StartFragment-->
<br />
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">MyDynamoDb</span><span lang="en-US" style="color: #686868;">:</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">Type</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">AWS::DynamoDB::Table</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">Properties</span><span lang="en-US" style="color: #686868;">:</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeDefinitions</span><span lang="en-US" style="color: #686868;">:</span></div>
<div style="color: #686868; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
- </div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeName</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"project"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeType</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"S"</span></div>
<div style="color: #686868; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
- </div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeName</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"owner"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeType</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"S"</span></div>
<div style="color: #686868; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
- </div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeName</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"votes"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeType</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"N"</span></div>
<div style="color: #686868; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
- </div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeName</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"members"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeType</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"N"</span></div>
<div lang="en-US" style="color: #894d34; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="color: #894d34; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868; font-size: 9pt;"> </span><span lang="en-US" style="color: #569cd6; font-size: 9pt;">KeySchema</span><span lang="en-US" style="color: #686868; font-size: 9pt;">: </span></div>
<div style="color: #686868; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
- </div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeName</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"project"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">KeyType</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"HASH"</span></div>
<div style="color: #686868; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
- </div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">AttributeName</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"owner"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">KeyType</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"RANGE"</span></div>
<div lang="en-US" style="color: #894d34; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="color: #894d34; font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868; font-size: 9pt;"> </span><span lang="en-US" style="color: #569cd6; font-size: 9pt;">ProvisionedThroughput</span><span lang="en-US" style="color: #686868; font-size: 9pt;">: </span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">ReadCapacityUnits</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"5"</span></div>
<div style="font-family: Menlo; font-size: 9.0pt; margin: 0in;">
<span lang="en-GB" style="color: #686868;"> </span><span lang="en-US" style="color: #569cd6;">WriteCapacityUnits</span><span lang="en-US" style="color: #686868;">: </span><span lang="en-US" style="color: #894d34;">"5"</span></div>
</blockquote>
<div>
<br /></div>
<div>
If you refer to the official class, <a href="https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_dynamodb/Table.html">https://docs.aws.amazon.com/cdk/api/latest/python/aws_cdk.aws_dynamodb/Table.html</a> you'll see that there is no Attribute Definitions... you're basically just given two parameters, "partition_key" and "sort_key", and that's it. So... what do you do, how do you define other Attributes?</div>
<div>
<br /></div>
<div>
<b>IMPORTANT</b>: This is just an example that I'm using to explain the syntax, please don't get into the philosophy of whether we need to be defining attributes while creating a DynamoDB table.</div>
<div>
<br /></div>
<div>
Here's the "trick". You basically need to find the python class that matches each of your properties. In this case, we're looking for a <b>Key Schema</b> and <b>Attribute Definitions</b>, and the corresponding classes are <span style="background-color: #1e1e1e; color: #d4d4d4; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 12px; white-space: pre;">KeySchemaProperty</span> and <span style="background-color: #1e1e1e; color: #d4d4d4; font-family: "menlo" , "monaco" , "courier new" , monospace; font-size: 12px; white-space: pre;">AttributeDefinitionProperty</span>. Once you've figured this out, you need to "guess" how to form the entire syntax around it with no documentation to guide you. It would look something like this (make sure to note which are VARIABLES, and which are ARRAYS):<br />
<br /></div>
<div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from aws_cdk import (<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> core,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> aws_s3 as s3,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> aws_dynamodb as ddb,<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span lang="EN-US">…</span><o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span lang="EN-US"><br /></span></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span lang="EN-US">MyDynamoDB</span> = ddb.CfnTable(<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> self, "MyDynamoDB",<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> table_name="dynamodb-prod-unicorn",<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> key_schema=[<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ddb.CfnTable.KeySchemaProperty(attribute_name="project",key_type="HASH"),</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ddb.CfnTable.KeySchemaProperty(attribute_name="owner",key_type="RANGE")</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ],<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> attribute_definitions=[</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ddb.CfnTable.AttributeDefinitionProperty(attribute_name="project",attribute_type="S"),</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ddb.CfnTable.AttributeDefinitionProperty(attribute_name="owner",attribute_type="S"),</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ddb.CfnTable.AttributeDefinitionProperty(attribute_name="votes",attribute_type="N"),</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ddb.CfnTable.AttributeDefinitionProperty(attribute_name="members",attribute_type="S"),</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">],</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> provisioned_throughput=ddb.CfnTable.ProvisionedThroughputProperty(<o:p></o:p></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">read_capacity_units=5,</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">write_capacity_units=5</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> )</span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> )</span><span style="font-family: "calibri" , sans-serif; font-size: 12pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="font-size: 12pt; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "times" , "times new roman" , serif;"><br /></span></div>
<div class="MsoNormal" style="margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "times" , "times new roman" , serif; font-size: 12pt;">This looks very complex, but it's actually quite simple. Also have in mind that this is ONLY in case you need to use a Property not supported by an official CDK construct, in this case </span><span style="font-family: "times" , "times new roman" , serif;"><a href="https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-dynamodb.Table.html">aws-cdk_aws-dynamodb.Table</a> .</span><br />
<span style="font-family: "times" , "times new roman" , serif;"><br /></span>
<br />
<div style="font-size: 12pt;">
<span style="font-family: "times" , "times new roman" , serif;"><br /></span></div>
</div>
</div>
<div>
<span style="font-family: "times" , "times new roman" , serif;">Let me know in comments if this helped.</span></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-50593124864072301762019-12-01T22:48:00.004+01:002019-12-01T23:01:06.083+01:00How failing VCDX changed my life<!--StartFragment-->
<br />
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I started
my professional career in 2003, in a NOC (Network Operations Center), making
sure that Network and Security services of the VIP customers of the ISP
(Internet Service Provider) ran smoothly. Till 2013, I mostly worked on big
Network and Security design implementations for big European and Middle East
companies, on Cisco and Juniper equipment, I got all the technical
certifications (including CCIE). I jumped into Software Defined world and
Virtualization as soon as it hit the market, and got my VCIX (VMware Certified
Implementation Expert) Certification on Network Virtualization track (NSX,
basically) soon after VMware acquired Nicira in 2014. </div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
My point
is that I did so many technical designs and implementations, mostly in Data
Center environments, that when I found out about VCDX (VMware Certified DESIGN
Expert), I was sue that I'd get it in the first attempt.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
That did
not happen. Not only that it didn't happen, but I faced one of the toughest
wake up calls… ever!</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
What's
VCDX all about?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span lang="en-US">VCDX
is currently the most prestigious VMware Certification, held by less than 300
people in the world (<a href="https://vcdx.vmware.com/" target="_blank">check out the official directory</a></span><span lang="en-US">). VCDX is NOT a
technical certification. Even so, you need to be a SME (Subject Matter Expert)
in all the areas that your Design covers.</span></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
VCDX is
all about the community. The official VMware documentation and support is all
right, I guess, but the community is one of the greatest I've ever stumbled
upon. I'd actually like to seize this opportunity to thank some of the people
who helped me while I was working on my design, and doing my mocks (mock is a practice design defence):</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li><a href="https://lucacamarda.blog/about/" target="_blank">Luca Camarda</a>, my VCDX mentor.</li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Joe Silvagi, VCDX #175, for
organizing amazing VCDX Workshops</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><a href="https://twitter.com/greggrobertson5?lang=en" target="_blank">Gregg Robertson</a> </span><span style="font-family: "calibri"; font-size: 11.0pt;">for maintaining the VCDX
Slack Channel</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><a href="https://twitter.com/do0dzZZ" target="_blank">(Abdullah)x2</a>, <a href="https://twitter.com/zecevicigor" target="_blank">Igor Zecevic</a>,
<a href="https://twitter.com/MannySidhu2" target="_blank">Manny Sidhu</a>, <a href="https://twitter.com/uprightvinyl" target="_blank">Chris Porter</a> and <a href="https://twitter.com/Apollokre1d" target="_blank">Kiran Reid</a> for so many mocks and defence
prep sessions</span></li>
</ul>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
About my
journey</h2>
<h3 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Phase 1:
Delusion</h3>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I came
into a VCDX as a "great architect", and received such a slap of
reality, understanding how much I actually didn't know. Yes, it's all about the
WHY, and no matter how many times you hear this, you just don't get it until
you start presenting your design to other VCDXs and candidates. </div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I learned
SO MUCH about what good design needs to cover, and why it's so important that
you understand how all the business requirements map into technical
requirements, which drive 100% of your design decisions. I understood that if
you have a Design Decision that you can't explain how you reached (what were
the options, and how one of the requirements triggered that one, covering all
the risks that it introduces and how you mitigate them) - you will simply fail.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If I can
give you one advice, it's - be patient, and accept tips other VCDXs are giving
you. They're not being mean, they really want you to pass.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Phase 2:
Success</h3>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
During
VMworld 2018, I had a mock session where I presented my VCDX design to 6 VCDXs.
It was brutal… The design I thought was bulletproof got destroyed to pieces.
Result: I learned so much… It took me some time to recover from this
experience, but once I did, I understood what an amazing privilege it was to
actually have everyone listen to me for over an hour. This is when my mind just
"clicked". I changed my design, and was approved for defence in the
next available defence slot.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I had to
change my design twice before it was approved for defence. I never got to
defend it, but I still consider the journey a success for the following
reasons:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">I learned so much about what
great design is. I switched to a complete different set of technologies
(public cloud mostly), but everything I learned still applies, 100%, and I
use it daily building Cloud architectures.</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">I learned hot to be more
down-to-earth, and that I'm not a great architect. I still have so much to
learn.</span></li>
<li lang="en-US" style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">I understood what "it's
all about WHY" means. Technology needs to help achieve business
goals, being technically superior doesn’t mean you can make great designs.</span></li>
</ul>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Phase 3:
Change of plans</h3>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I ended
up changing jobs a few months before the defence date, which was unfortunate,
because I had to focus on an entire new set of technologies that I didn’t have
time to prepare my defence. Some may see this as a pity, after all the time and
effort I invested, but to be honest - I don’t see it that way. I see it as a
success story, as I still get to apply everything I learned… just on a
different set of technologies.</div>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Conclusion</h2>
<div lang="en-US" style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I highly
recommend you to go for VCDX. You might not get your number, you might get
crushed while getting the design right, but I guarantee one thing - you will
learn a lot, and became a much better architect then you currently are.</div>
<!--EndFragment--><br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-22537934224210720782019-07-04T16:17:00.000+02:002019-10-17T16:17:58.137+02:00What is Service Mesh and do I need one?<h3>
Let's start with this - what problem does Istio solve?</h3>
<div>
Or... are we just using it cause it's the next cool thing? To be honest - there's a bit of both, from what I'm seeing with most of my customers.</div>
<div>
<br /></div>
<div>
To illustrate the problem Istio solves, lets take an example customer who already has their Kubernetes clusters. It doesn't matter on which cloud/data center k8s Masters and Workers are. SRE team is properly skilled, and operating the environment. The Developers see the great improvement as there's a clear improvement, plus an SRE team clearly understands what they need. Life is good.</div>
<div>
<br /></div>
<div>
Knock, Knock...</div>
<div>
<br /></div>
<div>
D: Oh... it's Marketing! Hello, Marketing, how can we help you?</div>
<div>
M: Hey, Developers!!! How are you, bros? So... we have this super awesome new feature we'd like to test only in Southern Spain, and only on iPhones... and maybe just like half of the users if possible. How long do we need to do this?</div>
<div>
D: ˆ%#$%!@ say... what??? Hey @SRE team, any chance Kubernetes can manage traffic management this granular?</div>
<div>
SRE: Hmmm... How many people are we allowed to add to the team to operate the environment? Does potential business benefit justify contracting new SREs?</div>
<div>
<br /></div>
<h3>
Enter - Service Mesh</h3>
<div>
And let's consider Istio, as my favourite Service Mesh at the moment...</div>
<div>
<br /></div>
There are 3 Core Features of Istio:<br />
<br />
<ol>
<li><b>Traffic Management</b>: We can do Canary Testing, where we would like redirect 10% of traffic to the New version of the app. Or, create an algorithm that redirects an application to a different version, such as - iPhone users, let me route you over ... here.</li>
<li><b>Security Authentication and Authorization</b>: Identity is assigned to each Pod when it's spun up, and we can create rules and policies for ACL, to say what services they can access.</li>
<li><b>Logging</b>: Istio also has a dashboard in Grafana.</li>
</ol>
<br />
<br />
<b>Istio is a CONTROL PLANE</b> (adds a pluggable Control Plane), and a Service Mesh is an actual Data Plane. Everything that Istio does is via Envoy Proxy, which is a literal Sidecar that is spun up with EACH Kubernetes Pod.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blog.aquasec.com/hs-fs/hubfs/Blog/Istio%20No.%203/Image1.png?width=1000&name=Image1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="585" data-original-width="800" height="291" src="https://blog.aquasec.com/hs-fs/hubfs/Blog/Istio%20No.%203/Image1.png?width=1000&name=Image1.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h3>
What are some elements in the Istio architecture diagram above?</h3>
<h4>
Pilot</h4>
Delivering config to the Proxies (Envoy). As a User you interact with the Pilot, through CLI, Automatically, or CI/CD. Pilot is in charge of:<br />
<span style="white-space: pre;"> </span>- Service Discovery<br />
<span style="white-space: pre;"> </span>- Intelligent routing<br />
<span style="white-space: pre;"> </span>- Resiliency<br />
<br />
<h4>
Envoy Proxy</h4>
L7 Load Balancer, Sidecar for all the Containers. It's a literal Sidecar, and Envoy Proxy is deployed along with EACH of the Pods. It takes care of:<br />
<span style="white-space: pre;"> </span>- Dynamic Service Discovery<br />
<span style="white-space: pre;"> </span>- Load Balancing<br />
<span style="white-space: pre;"> </span>- TLS Termination<br />
<span style="white-space: pre;"> </span>- Health Checks<br />
<span style="white-space: pre;"> </span>- Staged Rollouts<br />
<br />
<h4>
Mixer</h4>
Access control, quota checking, policy enforcement. Mixer keeps checking and getting reports if all Proxies are alive and well. Single API for syndicating, so Plugins for Monitoring, API management or Prometheus would go to Mixer.<br />
<br />
<h4>
Citadel</h4>
Strong service-to-service and end-user authentication with built-in identity and credential management.<br />
<br />
<h4>
Istio CA</h4>
Handles the certificates, to secure the communications.<br />
<br />
Istio uses the following configuration concepts:<br />
<span style="white-space: pre;"> </span>- Virtual Service<br />
<span style="white-space: pre;"> </span>- Destination Rule<br />
<span style="white-space: pre;"> </span>- Gateway<br />
<span style="white-space: pre;"> </span>- Service Entry<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://i.postimg.cc/76gWbDQY/istio2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="524" data-original-width="665" height="252" src="https://i.postimg.cc/76gWbDQY/istio2.png" width="320" /></a></div>
<br />
<br />
This entire mechanism seems (and is) pretty complex, but it allows us so much more in the micro service architecture. For more details <a href="https://istio.io/docs/reference/config/" target="_blank">I recommend checking out the official documentation, it's pretty well organized and technically written.</a><br />
<br />
<h3>
Conclusion</h3>
<div>
Kubernetes as such adds a big operational overhead. Istio adds even more overhead, and a big complexity on top of your platform. Should you use Istio then? If you have a huge Kubernetes clusters, bunch of Cloud Native Applications designed with micro services, with hundreds... maybe thousands of containers, and you also have a business requirement that justifies adding the overhead - sure, Istio is awesome! If not... maybe look for a simpler solution to your problem.</div>
<br />
<div>
<br /></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-34748070332696254382019-04-06T11:28:00.000+02:002019-10-17T11:28:56.500+02:00I use API Gateway. Can I claim I have an API Strategy now?In the last few years, I've had the opportunity to talk to a number of customers who, when asked what their API strategy is, simply answer something like "We're using a NGINX as API Gateway", or "we got https://www.mulesoft.comlicenses, still struggling to implement all we need".<br />
<br />
Let's start like this: API Gateway is NOT an API Manager. It's… just a Gateway for your APIs. What does API Gateway do? API Gateway if your frontend. It manages the API requests.<br />
It enforces policies (AAA), and lets you manage your L7 Ingress… but there's so much more to a API Management to that.<br />
<br />
<h3>
How do you create your requirements?</h3>
When you design the API Management solution, you need to think about how to design a strategy for your particular business. To be specific, you need to focus on two things:<br />
<br />
<ul>
<li>Your Developers</li>
<li>Your Customers</li>
</ul>
<br />
Why?<br />
If you motivate your Developers to explore the ways to improve the APIs, and cross-reference this with Analytics capabilities, in order to achieve the continuous feedback of how your Customers are consuming, and how they'd like to consume your app. This means:<br />
<br />
<ul>
<li>You want to give the best APIs to your developers, so that they can achieve the best value.</li>
<li>You need to establish the API Team, who would be in charge of all your APIs, making sure that Usability and Security are of the highest quality.</li>
</ul>
<br />
<br />
<h3>
What do I need to build an API strategy?</h3>
<div>
You need to be sure you have all of the following aspects "covered":</div>
<br />
<ul>
<li>Developer Portal: Where you can quickly engage your developers and partners.</li>
<li>Analytics: to gain the deep insight into API usage and performance.</li>
<li>Operations Automation: Scale APIs at web scale with operational control.</li>
<li>API Development: Tools that help develop, version, deploy and monitor APIs.</li>
<li>Security, covering all the aspects of your APIs.</li>
<li>Monetization enablement: Setting up pricing rules, based on usage, load and functionality, issuing invoices and collecting payments.</li>
</ul>
<br />
<br />
<h3>
What are some API Management products that I should consider?</h3>
I personally prefer MuleSoft, probably due to the experiences in the past, but as it sometimes happens - Gartner doesn't fully agree with me. Here's what they've determined for 2019. What do you think?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://i.postimg.cc/mgcDmwpW/APImgmt.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="800" height="400" src="https://i.postimg.cc/mgcDmwpW/APImgmt.png" width="400" /></a></div>
<br />
<br />
<br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-34112147764117295072019-02-26T16:38:00.002+01:002019-02-26T16:39:11.588+01:00Kubernetes Proxy: Envoy vs NGINX vs HA Proxy<!--StartFragment-->
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Having spent quite
some time with Linux and Kubernetes admins, I've come to realize that
networking isn't one of their strong sides. Being a network guy myself, I feel
obliged to share my views on topics as important as this one. So, which proxy should you use in your Kubernetes cluster?</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Lets start with some
facts:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">All three of these proxies
are highly reliable, L7, proven proxies, with Envoy being the newest kid
on the block.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">All these proxies do an
outstanding job of routing traffic L7 reliably and efficiently, with a
minimum of fuss.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">There is no full parity of
features, but you can implement any critical missing features in the proxy
itself… the power of open source!</span></li>
</ul>
<div>
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div>
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div>
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s3-eu-west-1.amazonaws.com/matscloud-images/cdn-haproxy-k8s-f4fdae1769e2552f040fdac945982659.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="544" data-original-width="800" height="217" src="https://s3-eu-west-1.amazonaws.com/matscloud-images/cdn-haproxy-k8s-f4fdae1769e2552f040fdac945982659.png" width="320" /></a></div>
<div>
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
To keep the post
structure, just a few lines about each of these 3 Proxies:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;"><a href="https://www.haproxy.org/" target="_blank">HA Proxy</a></span><span style="font-family: "calibri"; font-size: 11.0pt;"> is the default Load Balancer
when it comes to Kubernetes. It was initially released in 2006, when the
Internet operated very differently than today, ergo… there's an issue of
slow adoption of new features. This is very serious when you consider
SECURITY, like support for last SSL/TLS versions.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;"><a href="https://www.nginx.com/" target="_blank">NGINX</a></span><span style="font-family: "calibri"; font-size: 11.0pt;"> is a high-performance web
server, FASTER and more modern then HA Proxy Load Balancer, WAF and so
many other things… and if you check out the SDN integrations (Cisco ACI,
VMware NSX, Nokia Nuage), these are all based on open source version of
NGINX. NGINX open source has a number of limitations, including limited
observability and health checks, so it comes down to what you're looking
for. If you want an enterprise product, depending on your company
environment - go with NGINX Plus, ACI or NSX (be sure to ask for -T). </span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11pt;"><b><a href="https://www.envoyproxy.io/" target="_blank">Envoy</a></b></span><span style="font-family: "calibri"; font-size: 11.0pt;"><b><a href="https://www.envoyproxy.io/" target="_blank"> Proxy</a></b> is new… so not very
mature, BUT - most modern, and used in production in Apple, Google among
others. Envoy was designed from the ground up for microservices, with
features such as hitless reloads, resilience, and advanced load balancing,
plus - and e</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">xposing dynamic APIs for configuration</span><span style="font-family: "calibri"; font-size: 11.0pt;">. THIS is a big deal, in the
world where proxies have been configured using static configuration files
(Envoy also supports static config, of course). And lets not forget that <b><a href="https://istio.io/" target="_blank">Istio Service Mesh</a></b>, which I'm a big fan and contributor of, </span>uses an extended version of the Envoy proxy.</li>
</ul>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-72255652054343290902018-11-20T12:15:00.000+01:002018-11-20T12:30:53.997+01:00How I prepared for AWS SA Professional examLast week I managed to pass the <b>AWS Solution Architect professional certification exam</b>. Here's my certification, in all its glory:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://i.postimg.cc/YCmt4BvT/Screenshot-2018-11-20-12-12-54.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="614" data-original-width="800" height="306" src="https://i.postimg.cc/YCmt4BvT/Screenshot-2018-11-20-12-12-54.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
If you've been following my blog, you'll know that I passed a Google Cloud Professional Architect exam in March. I wrote a few blog posts about how I prepared it, and you may find it all <a href="https://matscloud.blogspot.com/2018/03/how-i-passed-google-certified.html">here</a>.<br />
<br />
Even though I've been preparing for the AWS exam for quite a while, the two main reasons I went for GCP professional level exam first are simple:<br />
<br />
<ul>
<li>I think Google Cloud is a sleeping giant, and I wanted to be among the first certified experts. </li>
<li>AWS has much more services. For a professional level exam you don't just need to know some of them in depth, you need to know ALL of them in depth, in order to make the right architecture that fits the customers requirements.</li>
</ul>
<br />
<br />
<h3>
How I prepared</h3>
Simple:<br />
<br />
<ul>
<li><a href="https://linuxacademy.com/amazon-web-services/courses">Linux Academy</a> has amazing hands-on courses for both Associate and Professional level. In my experience - the only one that really prepare you for this exam.</li>
<li>Work experience. This is where it gets tricky… AWS has a wide service catalogue, and your work environment hands is unlikely to cover the entire blueprint.</li>
</ul>
<br />
<br />
<h3>
Difference between AWS Associate and Professional level exams</h3>
This is something I get asked a lot. Here is the main difference:<br />
<br />
<ul>
<li>To pass the associate level exam, you need to know what each service does. The questions are straight forward, if you know what the service does - you'll eliminate most of the options in your test, and get the right answer.</li>
<li>AWS SAP (Solutions Architect Professional) is a real world business problem oriented exam. It's understood that you know all the AWS Service Catalogue in depth, and you are tasked with getting the most optimal architecture based on the customer requirements. You will get 77 different business scenarios (this is a LOT of text, so be prepared), and each one has 4-5 possible answers, which are all correct, you just need to figure out which one is the best for that particular scenario.</li>
</ul>
<br />
<br />
This basically means that if the question is how to connect your VPN with your on-premises infrastructure in the most cost efficient way, the answer will vary:<br />
<br />
<ul>
<li>In Associate level, you will go with VPN IPSec, cause Direct Connect is more expensive.</li>
<li>In Professional level you'll have to go deeper, and it's likely that mapping the use case with the architecture, Direct Connect could come out as the most cost efficient option.</li>
</ul>
<br />
<br />
<h3>
AWS vs GCP professional certifications</h3>
This is a tricky one… Basically this is how it is:<br />
<br />
<ul>
<li>GCP exam is very, very difficult. I feel like it's a Cloud Architect and DevOps merged into one exam, which makes it quite complex and "uncomfortable" at moments. BUT - GCP doesn’t have nearly as many services as AWS does in the Service Catalogue, so I guess the blueprint is narrower, which kind of justifies the complexity of the exam.</li>
<li>AWS is difficult, and long, requires high concentration during the 170 minutes, and probably what I like more - tests you for the real world skills. You will potentially get the same possible architectures as the answers in many different questions, and I feel it's impossible for someone to pass it even if they knew the questions, you really need an architect mind. On the positive side - there are no trick questions, so if you're good - you'll pass, it's as simple as that.</li>
</ul>
<br />
<br />
<h3>
What's next? </h3>
I'm going all in for my <b>VMware VCDX</b> (Design Expert) exam now. Did the design, going for the defence. I think I'm in the point in my career to go for something like this, get roasted for thinking I'm a super architect… Bring it on, my ego is about to be destroyed, but I feel like I'll come out of the experience as a true business architect.Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-64486008251996797972018-11-16T14:29:00.000+01:002019-08-12T16:36:16.931+02:00On relevance of CCIE in 2019A question I've been getting a lot from the Network Engineers, should they go for CCIE. There are two points to this question:<br />
<br />
<ul>
<li>Knowledge and skill</li>
<li>Value of CCIE as a Certification</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://www.ciscozine.com/wp-content/uploads/How-many-CCIEs-are-in-the-world.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="180" data-original-width="310" src="https://www.ciscozine.com/wp-content/uploads/How-many-CCIEs-are-in-the-world.jpg" /></a></div>
<br />
<br />
Let me get into more detail.<br />
<br />
<h3>
Value of CCIE as gaining skill and knowledge</h3>
Networking as such is changing. A network engineer for the cloud era needs to understand programmability, APIs, SDN with its use cases, Public Cloud networking (inter and intra public cloud). BUT, if you've ever talked to a network engineer who doesn't come from hardcore cisco or juniper networking, and rather comes from systems (VMware or Linux), or someone who's just studied something like OpenFlow and considers hardware to be a "commodity", you'll notice how due to lack of basic networking L1-4 concepts, they tend to not understand some limitations in both functionality and performance. There are exceptions, of course, and I want to acknowledge that!!! The point I'm trying to make is that CCIE gives you the best of breed base for any kind of programmable, cloud, Kubernetes or whichever networking-related activity you want to pursue in the future.<br />
<br />
<h3>
Value of CCIE as a Certification</h3>
This is a completely different topic. If you want to do your CCIE just because you want more money from your employer - don’t. Go learn AWS, learn Python and Ansible, maybe some ACI and NSX but from the "north side" (API). The days when getting a CCIE meant an immediate salary increase of 50% are over… It is now a step in your trip, not the final goal.<br />
<br />
<h3>
Conclusion</h3>
Should you go for a CCIE? Yes. If you are serious about networking, you 100% should. You will learn all that other SDx and Cloud stuff much more easy if you understand bits and bytes. Hey, I passed my Google Cloud, AWS, and NSX highest level technical certifications greatly thanking to the networking knowledge I learned working on the field as a CCIE... I'm just doing Networking in a different way now. But - it's still networking, L2 and L3, same old MAC, IP and BGP, just consumed in a different way.<br />
<br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com1tag:blogger.com,1999:blog-6091645117172561542.post-32364604732003627712018-10-29T09:16:00.003+01:002018-10-29T09:16:37.388+01:00Just married: IBM and RedHat. What does this mean for Cisco and VMware Multi-cloud offer?As per yesterdays announcement, IBM is acquiring Red Hat in deal valued at $34 billion (more about this <a href="https://www.cnbc.com/2018/10/28/ibm-to-acquire-red-hat-in-deal-valued-at-34-billion.html">here</a>). This is another one in a row of deals I did not expect to happen:<br />
<br />
<ul>
<li>Oracle acquired Sun Microsystems</li>
<li>Microsoft acquired GitHub</li>
<li>Dell acquired VMware</li>
</ul>
<br />
<br />
How disruptive can a Purple Hat really be? VMware survived being acquired by Dell quite well... will RedHat have the same luck, or not? What I know for sure is that the RedHat employees are panicking right now...<br />
<br />
Sure, 3k billion is a big sum, but also a bold move by IBM on the conquest to the Multi-Cloud market. Combined we're looking at (to name a few):<br />
<br />
<ul>
<li>Ansible for the Automation</li>
<li>OpenShift, as the best of breed PaaS based on Kubernetes</li>
<li>CloudForms as a potential CMP (I wonder how this will work out...)</li>
<li>Watson for all AI/Machine Learning related</li>
<li>IBM Cloud as a Public Cloud platform</li>
</ul>
<br />
<br />
Is this a winner combo? Or do other Hybrid Cloud promoters, like Cisco and VMware have equally good lock-in-free proposals?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://cdn-images-1.medium.com/max/1200/1*E4crAzYRahH2y8S_ZjSiRw.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="340" data-original-width="800" height="168" src="https://cdn-images-1.medium.com/max/1200/1*E4crAzYRahH2y8S_ZjSiRw.jpeg" width="400" /></a></div>
<br />
<br />
As a Hybrid Cloud and DevOps advocate, and a European CTO, I've had the experience to "casually chat" to many European companies about their Cloud strategy. Two things are evident:<br />
<br />
<ul>
<li>The buyer is changing, Multi-Cloud is an APPLICATION strategy, not the infrastructure strategy (read more about this <a href="https://www.onug.net/blog/multi-cloud-is-an-application-strategy-not-an-infrastructure-strategy/">here</a>).</li>
<li>Companies don't really know who to trust, as what they're being told by various vendors and providers is not really coherent. This makes is pretty difficult to actually build a Cloud strategy (don't get me started on CEOs who'll just tell you "We've adopted Cloud First", and actually think they have a cloud strategy).</li>
</ul>
<br />
<br />
Due to all this:<br />
- IBM and RedHat, as software companies, will be able to get to the Application market.<br />
- Neither of the two can do Infrastructure as well as VMware & Cisco.<br />
<br />
<br />
<h4>
How important is this? Very! And here is why.</h4>
<b>Cisco</b> has:<br />
<br />
<ul>
<li>Cloud Center, a true application oriented micro-services ready CMP, Public Cloud and Automation Tool agnostic, equipped with the right Benchmarking and Brokering tools, that integrates quite well with the infrastructure, and workflow visibility platforms.</li>
<li>ACI and Tetration, that enable the implementation of coherent and consistent Network and Security Policy Model across multiple private and public clouds, along with the workload visibility.</li>
<li>HyperFlex and CCP, providing enterprise production-ready, lock-in-free Kubernetes solution on a Hyper Converged infrastructure.</li>
<li>AppDynamics and Turbonomic, a true DevOps combo for the Day 2 we're all fearing in the Cloud, letting the application architects model their post-installation architecture, and monitor the performance of each element, latency between different elements, and assure the optimal user experience.</li>
</ul>
<br />
<br />
<b>VMware</b> has:<br />
<br />
<ul>
<li>vRealize Automation, the best of breed Automation and Orchestration Hybrid Cloud ready platform.</li>
<li>PKS and VKE, KaaS platforms that provide the enterprise production-ready Kubernetes solution, with a fully prepared Operations component, in both - private and public cloud.</li>
<li>Wavefront, application visibility tool running on Containers, designed with Cloud applications and Micro Services in mind, with just insane performance.</li>
<li>NSX, including the full SDN stack in both, Data Center and Cloud, with probably the best API (both documentation and usage wise).</li>
<li>Partnership with AWS, Azure, GCP and IBM, to leverage the most demanded Hybrid Cloud use cases in a "validated design" fashion.</li>
</ul>
<br />
<br />
<br />
<h4>
What does this all mean?</h4>
Multi-cloud is still a space that, based on Gartner and IDC, over 90% of Companies are looking at. Big companies are making their moves... so just grab your popcorn, and observe. It's going to be a fun ride!Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-73031566643323886342018-08-30T08:28:00.000+02:002018-08-30T08:44:51.319+02:00Why SDN isn't where we thought it would beThe SDN hype started a few years ago. Everyone was talking about it as the next big thing, and it all made so much sense. I started exploring SDN while Nicira and Insieme were just two startups, and got even deeper into it when they were bought by VMware and Cisco and rose as ACI and NSX.<br />
<br />
SDN makes perfect sense. A single point of management and operations of the entire data center network, micro Segmentation as an embedded feature, REST API support for automation, possibility to move the workloads between Sites without having to reconfigure the security policy, and a bunch of others. It truly is a missing piece, arriving a bit too late. So… why hasn't the same happened like when we started using server virtualization? Why isn't everyone implementing these technologies, and celebrating the benefits while singing their favourite tune?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s33.postimg.cc/6oo8xi3m7/image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="351" data-original-width="353" height="318" src="https://s33.postimg.cc/6oo8xi3m7/image.png" width="320" /></a></div>
<br />
<br />
In my opinion, two reasons: misleading PowerPoints and vendors with the wrong go to market strategy.<br />
<br />
<h3>
Misleading PowerPoints</h3>
Networking tends to be more complex then Compute and Storage in the Data Center. You have a group of independent network devices that need to transfer an insane number of packets between different points, with zero latency, and no time to talk to each other and coordinate the decisions. When you introduce Automation into the equation, it all gets really interesting. With SDN we introduced overlay, and managed to somehow make all this easier. Where is the problem then?<br />
<br />
Automation is an awesome concept. If you automate, you will improve the delivery times, and always end up with the same results. Automation is not new… it's been here since the 70s, and even though the execution premises have changed, one thing stays the same:<br />
<span style="white-space: pre;"> </span>- If you automate, you will save a lot of time and resources.<br />
<span style="white-space: pre;"> </span>- To create automation, you need a lot of knowledge, experience and a lot of time and effort.<br />
<br />
The truth about misleading PowerPoints lies in the second point. Everyone rushed to explain to their customers how their SDN has an API, and how you can automate everything in an instance, I saw bonus hungry AMs and SEs singing the songs to the customers about how they can use the automation tool of choice. "There's an API so you're good bro!" Unfortunately, this is far from the truth. Yes, SDN supports automation of your network, but it takes a lot of hard work to set it up right, and if you sold something to the customer without setting their expectations right… well, he will be disappointed.<br />
<br />
What is the truth? Both ACI and NSX are mature solutions, but the SDN is no longer a group of independent switches, it needs to be integrated in the wider ecosystem, and it makes all the difference who integrates all this in your Data Center. If the customers were prepared for this from the beginning, I think we'd all bee seeing a whole lot more SDNs.<br />
<br />
<br />
<h3>
Vendor Strategy</h3>
I'll talk about 2 big ones here - VMware and Cisco. Have you noticed how these two vendors have the same number of production references in each moment? Like there is some kind of secret synchronisation behind the curtain. Ever wondered why that is?<br />
<br />
The truth is that both, ACI and NSX, are great products. Yes, GREAT! It's also true that a surprisingly small number of "SDN experts" out there understands HOW and WHY these products need to be introduced in the data center ecosystem, so a majority of the happy SDN customers that Cisco and VMware are referencing are kind of fake, meaning - yes, they are using the product, and yes, it's in production, but it is not used as SDN. Sure, Cisco has <a href="https://developer.cisco.com/">DevNet</a>, and VMware has <a href="https://code.vmware.com/home">VMware Code</a>, and these are both great initiatives, but they still lack a critical mass… both of them do. [if you don’t know what these are, I STRONGLY recommend that you stop reading this post, and go check out both these websites, they are AWESOME].<br />
<br />
<h4>
<b><span style="color: #3d85c6;">What is Cisco's mistake?</span></b></h4>
Cisco counts on their traditional big partners to deliver ACI. These guys can sell networking to a networking department, they get BGP and VxLAN, they can build the fabric in what Cisco brutally named "a networking mode", and they can train the networking department to use ACI. That’s it. So… what about automation, IAC (infrastructure as code), what about the developers who are actually the true buyer here, and they just need to provision some secure communication for their code? Well, I'm afraid there's nothing here for them, because neither a Cisco's networking partner not the customers networking department are able to configure and prepare the ACI for what these guys really need SDN for. Customer simply isn't getting what they paid for, and they are pretty vocal about it in the social networks, so the product gets bad marketing.<br />
<br />
And yes, there are companies out there (such as mine) who are able to implement ACI as a part of a Software Defined ecosystem and help customer build automation around it, but Cisco somehow still isn's seeing the difference, and is still promoting same old networking partners to the customers to implement their ACI. Oh well… let's hope Cisco starts to understanding this before it's late.<br />
<br />
<br />
<h4>
<b><span style="color: #6aa84f;">What is VMware's mistake?</span></b></h4>
NSX is an entirely different story. The problem isn't VMware's strategy, but rather - the buyer. SDN is still networking, so the buyer is a Networking Department, but… Networking guys don’t know VMware, they know Cisco and Juniper. On the other hand there are System Admins who are desperate to gain control over network and not depend on the slow networking departments, but… they lack advanced networking knowledge. So NSX, being a brilliant product as it is, ended up in no mans land. VMware did everything to promote NSX to Network experts, if you're a CCIE, like me, you can actually do NSX cert exams without doing the training, and NSX is easy to learn and understand, but still, not enough hype around it among network admins. So, what happened? Well, for now there are many implementations of NSX used the way System and Security experts are able to promote and manage it, Micro Segmentation with some basic networking, but not even close to the NSX full potential, and again - not used as an SDN.<br />
<br />
<br />
<h4>
<b>What about the other SDN vendors?</b> </h4>
There are a few worth mentioning: Nokia Nuage, Juniper Contrail, some distributions of OpenDayLight (HP, Dell, Huawei, Ericsson, NEC, etc.). Two things are happening with these guys:<br />
<br />
<ul>
<li>Due to all mentioned above, the Customers are under the wrong impression that not even ACI and NSX are fully mature and stable solutions… If Cisco and VMware aren't able to invest what it takes and make it stable, what do you expect from the others?</li>
<li>In one moment all these guys made huge investments in their technology, and there was still no sales to support the investment, so - they lowered the prices and started selling the solutions that weren't yet mature. This caused customers dissatisfaction, and the rumor on the market that SDN just "isn't there yet". They can still recover… as long as they actually invest in product development and engineering skills, and let product sell itself. </li>
</ul>
<br />
<br />
<br />
<h3>
What should we expect in the next 2-4 years?</h3>
SDN is here to stay, even more so with IoT and Containers with a whole set of new micro Segmentation and Network automation requirements. It just takes it longer then anticipated to find it's place. I think the customers are slowly starting to get the non-planned effort to actually move from installing the SDN product - to using it as a Software Defined technology, which is good, so if you're considering SDN as a potential career path - add some automation and programming skills, and you're on the right track.Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com2tag:blogger.com,1999:blog-6091645117172561542.post-12608251176158036802018-08-29T14:23:00.002+02:002018-08-29T14:28:37.302+02:00Migrate HyperFlex Cluster to a new vCenterGet ready to have your mind blown. One of the easiest procedures I've encountered. You just need to follow these 3 steps, to migrate the entire HyperFlex vSphere Cluster with all its hosts from vCenter 1 to vCenter 2.<br />
<br />
<h3>
Before you start:</h3>
- Your environment might be different. I'm not responsible if something goes wrong, you're welcome to look for the official guides. I've tested it to migrate from vCenter 6.0 to 6.7 in August 2018.<br />
- VDS WILL NOT be migrated automatically, BUT - you can Export it into ZIP from the old vCenter, and import into the new one AFTER you've done all these steps, and the Uplinks will be automatically mapped. Be sure to include all the configuration, portgroups and all, both when you export and import.<br />
<br />
<h3>
Step 1.</h3>
Deploy your vCenter Server Appliance. I'll asume you're setting the standard username, administrator@vsphere.local<br />
<br />
<h3>
Step 2.</h3>
Create both Datacenter and Cluster in the empty new vCenter. For the ease of migration, use the same names. Connect all ESXi hosts from HyperFlex to the new Cluster. Just accept re-assigning of the licence, and wait to see the new host as Connected.<br />
<br />
<h3>
Step 3. </h3>
Re-register the Cluster to a new vCenter. I recommend that you observe the new vCenter in the background, so that you can follow the progress. To do this you need to SSD into your HyperFlex, and execute the following command (set your own parameters, of course):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">stcli cluster reregister --vcenter-cluster CLUSTER_NAME --vcenter-datacenter DATACENTER_NAME --vcenter-password 'NEW_vCENTER_PASSWORD' --vcenter-url NEW_vCENTER_IP --vcenter-user administrator@vsphere.local</span><br />
<br />
You will get this message:<br />
<br />
Reregister StorFS cluster with a new vCenter ... [this is where you wait for approx 10 minutes]<br />
Cluster reregistration with new vCenter succeeded<br />
<br />
<br />
<h3>
Additional Step:</h3>
If you are using VDS, this is when you need to import them to the new vCenter.<br />
<br />
<br />
And - you're done! Let me know in the comments if it worked as easy as this.Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com1tag:blogger.com,1999:blog-6091645117172561542.post-38627690104571552242018-06-08T19:46:00.001+02:002018-12-24T09:16:34.684+01:00Install PowerCLI on Mac, start using PowerNSXThis is something I've been wanting to publish for a while, and finally my Mac got formatted (no questions will be taken at this point...) and I had to re-install it all, and I just couldn't find the instructions on how to do it without just having to read pages and pages of disclaimers and stuff...<br />
<br />
<b>Why PowerCLI?</b> Cause it's a simplest way to automate your vCenter tasks, via the command line, fast and furious. Sure, one day a working vCenter web plugin will come, but who knows when...<br />
<br />
<b>Why PowerNSX?</b> Same... but for the NSX admins. Trust me, my life got so much better the day I stopped depending on vCenter Web GUI.<br />
<br />
How do I install and start using it? Simple. Just follow this 5 Steps guide...<br />
<br />
<br />
<h3>
<b>Step 1: Install PowerShell (check the update below first!!!)</b></h3>
Make sure you have GitHub:<br />
<br />
# git<br />
<br />
Clone the PowerShell installation package from GitHub:<br />
# <b>git clone --recursive https://github.com/PowerShell/PowerShell</b><br />
<br />
Once you got it, enter the Folder, and install the package (you'll be asked for a Password a few times):<br />
Submodule path 'src/libpsl-native/test/googletest': checked out 'c99458533a9b4c743ed51537e25989ea55944908'<br />
<br />
<br />
MatBook-Pro:~ mjovanovic$ <b>cd /Users/mjovanovic/PowerShell</b><br />
MatBook-Pro:PowerShell mjovanovic$ <b>./tools/install-powershell.sh</b><br />
<br />
Get-PowerShell Core MASTER Installer Version 1.1.1<br />
Installs PowerShell Core and Optional The Development Environment<br />
…<br />
Run "pwsh" to start a PowerShell session.<br />
*** NOTE: Run your regular package manager update cycle to update PowerShell Core<br />
*** Install Complete<br />
<br />
<br />
MatBook-Pro:PowerShell mjovanovic$ <b>pwsh</b><br />
PowerShell v6.0.2<br />
Copyright (c) Microsoft Corporation. All rights reserved.<br />
<br />
https://aka.ms/pscore6-docs<br />
Type 'help' to get help.<br />
<br />
PS /Users/mjovanovic/PowerShell><br />
<br />
You're in the PowerShell!!!<br />
<br />
<b><span style="color: red;">UPDATE</span></b>: As of December 2018, this method is no longer supported. You'd actually get into a quite "nifty" catch 22, where PowerShell 6.1.1 doesn't support most of relevant PowerCLI Commands, New PowerCLI doesn't support anything under 6.0.5, and some PowerNSX Commands require 6.0.1 and above. Awesome!<br />
<br />
<b><span style="color: blue;">SOLUTION</span></b>: <a href="https://github.com/PowerShell/PowerShell/releases/tag/v6.0.5" target="_blank">Download PowerShell 6.0.5, that one works!</a> Download it as a package, and install. The rest of the post remains the same.<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://s3-eu-west-1.amazonaws.com/matscloud-images/Screenshot+2018-12-24+09.14.53.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="796" data-original-width="640" height="400" src="https://s3-eu-west-1.amazonaws.com/matscloud-images/Screenshot+2018-12-24+09.14.53.png" width="321" /></a></div>
<br />
<br />
<h3>
<b>Step 2: Install PowerCLI</b></h3>
Now lets procede with the PowerCLI. More details on this Link, if you happen to need more details... but basically all you need is the following command: <a href="https://blogs.vmware.com/PowerCLI/2018/03/installing-powercli-10-0-0-macos.html">https://blogs.vmware.com/PowerCLI/2018/03/installing-powercli-10-0-0-macos.html</a><br />
<br />
<br />
PS /Users/mjovanovic/PowerShell> <b>Install-Module -Name VMware.PowerCLI -Scope CurrentUser</b><br />
<br />
Untrusted repository<br />
You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install<br />
the modules from 'PSGallery'?<br />
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): <b>Y</b><br />
<br />
<br />
<h3>
<b>Step 3: Install PowerNSX Modules</b></h3>
Ok, so now we just need to install the PowerNSX Modules:<br />
<br />
PS /Users/mjovanovic/PowerShell> <b>Find-Module PowerNSX | Install-Module -scope CurrentUser</b><br />
<br />
Untrusted repository<br />
You are installing the modules from an untrusted repository. If you trust this repository, change its InstallationPolicy value by running the Set-PSRepository cmdlet. Are you sure you want to install<br />
the modules from 'https://www.powershellgallery.com/api/v2/'?<br />
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "N"): <b>Y</b><br />
<br />
<b>Step 3.1: Resolve the Certificate Error:</b><br />
If you tried to connect to your vCenter now, you´d get this error:<br />
Connect-VIServer : 06/08/2018 18:32:43<span style="white-space: pre;"> </span>Connect-VIServer<span style="white-space: pre;"> </span>Error: Invalid server certificate. Use Set-PowerCLIConfiguration to set the value for the InvalidCertificateAction option to Ignore to ignore the certificate errors for this connection.<br />
<br />
Before Logging in to your vCenter, to avoid the Certificate problems (which you will most definitely have), first use, You need to set the Certificate Errors to FALSE:<br />
<br />
PS /Users/mjovanovic/PowerShell> <b>set-PowerCLIConfiguration -InvalidCertificateAction Ignore</b><br />
<br />
Perform operation?<br />
Performing operation 'Update PowerCLI configuration.'?<br />
[Y] Yes [A] Yes to All [N] No [L] No to All [S] Suspend [?] Help (default is "Y"): Y<br />
<br />
PS /Users/mjovanovic/PowerShell><br />
<br />
<br />
<h3>
<b>Step 4: Log into the NSX Manager and vCenter</b></h3>
Now you are GOOD TO DO, you can Log in to your NSX, and to the vCenter:<br />
<br />
PS /Users/mjovanovic/PowerShell> <b>Connect-NsxServer -NsxServer 10.20.70.18 -Username admin -Password M4TSCL0UD</b><br />
<br />
PowerNSX requires a PowerCLI connection to the vCenter server NSX is registered against for proper operation.<br />
Automatically create PowerCLI connection to 10.20.70.37?<br />
[Y] Yes [N] No [?] Help (default is "Y"): Y<br />
<br />
WARNING: Enter credentials for vCenter 10.20.70.37<br />
<br />
PowerShell credential request<br />
Enter your credentials.<br />
User: administrator@vsphere.local<br />
Password for user administrator@vsphere.local: **************<br />
<br />
Version : 6.4.0<br />
BuildNumber : 7564187<br />
Credential : System.Management.Automation.PSCredential<br />
Server : 10.20.70.18<br />
Port : 443<br />
Protocol : https<br />
UriPrefix :<br />
ValidateCertificate : False<br />
VIConnection : 10.20.70.37<br />
DebugLogging : False<br />
DebugLogfile : \PowerNSXLog-admin@10.20.70.18-2018_06_08_18_37_32.log<br />
<br />
<br />
<h3>
<b>Step 5: Start using PowerNSX</b></h3>
You can do so many things here! I recommend this Guide to get you started:<br />
<a href="https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-automating-vsphere-with-powernsx.pdf">https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmware-automating-vsphere-with-powernsx.pdf</a><br />
<br />
Most important command:<br />
<div class="p1">
<span class="s1">PS /Users/mjovanovic> </span><span class="s2">get-command</span><span class="s1"> </span><span class="s3">-module</span><span class="s1"> PowerNSX<span class="Apple-converted-space"> </span></span></div>
<div class="p2">
<span class="s1"></span><br /></div>
<div class="p1">
<span class="s1">CommandType <span class="Apple-converted-space"> </span>Name <span class="Apple-converted-space"> </span>Version<span class="Apple-converted-space"> </span>Source <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">----------- <span class="Apple-converted-space"> </span>---- <span class="Apple-converted-space"> </span>-------<span class="Apple-converted-space"> </span>------ <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxDynamicCriteria <span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxDynamicMemberSet<span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxEdgeInterfaceAddress<span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxFirewallExclusionListMember <span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxFirewallRuleMember<span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxIpSetMember <span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxLicense <span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxLoadBalancerPoolMember<span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxLoadBalancerVip <span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX <span class="Apple-converted-space"> </span></span></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
span.s2 {font-variant-ligatures: no-common-ligatures; color: #ebef18}
span.s3 {font-variant-ligatures: no-common-ligatures; color: #37ef20}
</style>
<br />
<div class="p1">
<span class="s1">Function<span class="Apple-converted-space"> </span>Add-NsxSecondaryManager<span class="Apple-converted-space"> </span>3.0.1118 <span class="Apple-converted-space"> </span>PowerNSX<span class="Apple-converted-space"> </span></span></div>
...<br />
<br />
Just play around with these!Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-11954507450110400852018-03-20T09:20:00.002+01:002018-05-19T19:47:13.038+02:00How I passed Google Certified Professional Cloud Architect ExamAfter a few months of heavy preps, I managed to pass the exam. I got the electronic certificate, and supposedly I'll get a Cloud Architect Hoodie! Yeah, I'm gonna wear it :)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk3SZDY-Ml1sB95RGKCs2-C0R_5Yyjwkn2Xu1AeHcd5jX-dPOuA0v6npodvVoE_WkvGEOw6WUYRvTIff4xfv4FGXVwg1NBVz4G80878nEoWKzVduc0gCl457IDXWHy_wf3EIYIV-NC1NVc/s1600/Screenshot+2018-03-19+09.50.49.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1194" data-original-width="1586" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk3SZDY-Ml1sB95RGKCs2-C0R_5Yyjwkn2Xu1AeHcd5jX-dPOuA0v6npodvVoE_WkvGEOw6WUYRvTIff4xfv4FGXVwg1NBVz4G80878nEoWKzVduc0gCl457IDXWHy_wf3EIYIV-NC1NVc/s320/Screenshot+2018-03-19+09.50.49.png" width="320" /></a></div>
<br />
<br />
The exam is every bit as difficult as advertised. I did A LOTS of Hands On in the Google Cloud Platform (the 300$ that Google gives you to play around comes in quite handy), without it I don't think it's possible to pass, bunch of questions have commands to choose from, and a heavy focus on App Development and Linux Commands. If you want to know how I prepared, check out my previous posts:<br />
<br />
<ol>
<li><a href="http://matscloud.blogspot.com/2018/02/google-cloud-architect-why-and-how-to.html">Why I decided to become a Certified Cloud Architect, why Google Cloud, and how I want to prepare</a></li>
<li><a href="https://matscloud.blogspot.com.es/2018/02/big-data-for-infrastructure-engineers.html">Introduction to Big Data and Hadoop</a></li>
<li><a href="https://matscloud.blogspot.com.es/2018/02/google-cloud-platform-gcp-how-do-i.html">Google Cloud - Compute Options (IaaS, PaaS, CaaS/KaaS)</a></li>
<li><a href="https://matscloud.blogspot.com.es/2018/03/google-cloud-platform-gcp-how-do-i.html">Google Cloud - Storage and Big Data Options</a></li>
<li><a href="http://matscloud.blogspot.com/2018/03/public-cloud-networking-and-security.html">Google Cloud - Networking and Security Options</a></li>
</ol>
<div>
<br /></div>
<br />
Stay tuned, my Cloud is about to get much more DevOps-y in 2018!Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com6tag:blogger.com,1999:blog-6091645117172561542.post-5284341898724213072018-03-04T15:48:00.000+01:002018-06-15T16:10:42.892+02:00Public Cloud Networking and Security: VPCs, Interconnection to Cloud, Load Balancing<!--StartFragment-->
<br />
<div style="border-width: 100%; direction: ltr;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 7.0423in;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 7.0423in;">
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I'm so happy to
finally be here, at the Networking part of the Public Cloud!!! I know, there
are more important parts of Cloud then Networks, but SDN is my true love, and
we should give it all the attention it deserves.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<b>IMPORTANT</b>: In this
post I will be heavily focusing on Google Cloud Platform. The concepts
described here apply to ANY Public Cloud. Yes, specifics may vary, and in my
opinion GCP is a bit superior to AWS and Azure at this moment, but if you
understand how this one works - you'll easily get all the others.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h1 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
Virtual
Private Cloud (VPC)</h1>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">VPC (Virtual Private Cloud)</span> provide global
scalable and flexible networking. This is an actual Software Defined Network
provided by Google. <span style="font-weight: bold;">Project can have up to 5 VPC
- Virtual Private Networks.</span> <span style="font-size: 11pt; font-weight: bold;">VPC can be global, and contains subnets and uses a private IP space. Subnets are regional. </span><span style="font-size: 11pt;">The network that you are provided with VPC
is:</span></div>
</div>
</div>
</div>
<br />
<div style="border-width: 100%; direction: ltr;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 7.0423in;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 7.0423in;">
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Private</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Secure</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Managed</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Scalable</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Can contain <b>up to 7000 VMs</b></span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Once you create the
VPC, you have a cross-region RFC1918 IP Space network, using Googles private Network
underneath. It uses the Global internal DNS, load balancing, firewalls, routes,
and you can scale rapidly with global L7 Load Balancers. <b>Subnets within VPC can
only exist within Region/Zone, you can't extend a Subnet over your entire VPC.</b></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
VPC Networks can be provisioned in:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Auto Mode</span><span style="font-family: "calibri"; font-size: 11.0pt;">, where the Subnet(s) is set (automatically assigned) in
every region. Firewall rules and routes are preconfigured.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Custom mode</span><span style="font-family: "calibri"; font-size: 11.0pt;">, where we have to manually
configure the subnets.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<!--StartFragment-->
</div>
</div>
</div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
IP
Routing and Firewalling</h2>
<div style="border-width: 100%; direction: ltr;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 7.0423in;">
<div style="direction: ltr; margin-left: 0in; margin-top: 0in; width: 7.0423in;">
<!--EndFragment--><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt; font-weight: bold;">Routes</span><span style="font-size: 11pt;"> are defined for the networks to which
they apply, and you can use them if you want to apply the route only for the
Instances with a certain "instance tag" (If you don't specify the
TAG, the route applies to all the instances).</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When you use the
Routes to/from the Internet, you have 2 options:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Many-to-one
NAT</span><span style="font-family: "calibri"; font-size: 11.0pt;">, multiple
hosts mapped to one public IP (known as SNAT in Cloud solutions such as OpenStack, <a href="http://matscloud.blogspot.com/2017/12/openstack-networking-explanation-for.html">check out my OpenStack post from some time ago for details about OpenStack Networking</a>).</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Transparent
proxies</span><span style="font-family: "calibri"; font-size: 11.0pt;">, that
direct all external traffic to one machine (Floating IPs per OpenStack terminology).</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
Project can contain various VPCs (<span style="font-weight: bold;">Google allows you to create up to 5 VPCs per project</span>). VPCs also have Multi Tenancy. All the resources in GCP belong to some VCP. Routing and Forwarding must be configured to allow traffic within VPC, and with the outside world. You also need to configure the <span style="font-weight: bold;">Firewall Rules</span>.</div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11pt; margin: 0in;">
VPCs are GLOBAL, meaning the Resources can span anywhere around the world. Even so, <span style="font-weight: bold;">instances from different regions CANNOT BE IN THE SAME SUBNET</span>. <span style="font-size: 11pt;">An instance needs to be in the same region as a reserved static IP address. The zone in the region doesn't matter.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Firewall Rules can be based on the Source IP (ingress)
or Destination IP (Egress) There are DEFAULT "allow egress" and
"deny ingress" rules, which are pre-configured for you, with the minimum priority (65535). </span>This means that if you configure the new FW rules with the lower number/higher priority, these will be taken into account, instead of the default ones.<span style="font-weight: bold;"> GCP Firewall
rules are STATEFUL.</span> You can also use <span style="font-weight: bold;">TAGs</span>
and <span style="font-weight: bold;">Service Accounts</span>
(something@developer.blabla.com for example) to configure the Firewall rules, and this is probably THE BIGGEST advantage of the Cloud Firewall, because you can do Micro Segmentation in a native way.
Once you create a Firewall Rule, a TAG is created, so the next time you create
an instance, and apply that rule, it will not create it again, just attach the
TAG to your instance.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There are 2 types of IP addresses in VPC:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- <b>External</b>, in the Public IP space</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- <b>Internal</b>, in the Private IP space</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
VPCs can communicate to each other using a Public IP
space (External networks visible on the Internet). <b>External IP can also be
ephemeral (change every 24 hours) or static</b>. VMs don't know what their external
IP is. <b>IMPORTANT</b>: If you RESERVE an External IP in order to configure it as STATIC, and not use it for an Instance
or a LB - you will be charged for it! Once you assign it - it's for free.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When you work with Containers -<span style="font-weight: bold;"> </span>containers need to focus on the
Application or Service. They don't need to do their own routing, it simplifies
the traffic management.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14pt; margin: 0in;">
Can I use a single RFC 1918 space within few GCP Projects?</h2>
<div>
<span style="font-family: "calibri"; font-size: 11pt;">Yes, using a </span><span style="font-family: "calibri"; font-size: 11pt; font-weight: bold;">Shared VPC</span><span style="font-family: "calibri"; font-size: 11pt;"> - Networks can be shared across
Regions, Projects etc. If you have different Departments that need to work on
the same Network resources, you'd create two separate projects for them, give the access only to the project they work on, and
use a single Shared VPC for the Network resources they all need to access.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;"> </span></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Google
Infrastructure</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Google's network
infrastructure has three distinct elements:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>Core data centers (central circule)</b>, used for
the Computation and Backend storage.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Edge Points
of Presence (PoPs)</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
Edge Points of Presence (PoPs) are where we connect Google's network to
the rest of the internet via peering. We are present on over 90 internet
exchanges and at over 100 interconnection facilities around the world.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Edge caching
and services nodes (Google Global Cache, or GGC)</span><span style="font-family: "calibri"; font-size: 11.0pt;">. Our edge nodes (called
Google Global Cache, or GGC) represent the tier of Google's infrastructure
closest to our users. With our edge nodes, network operators and internet
service providers deploy Google-supplied servers inside their network.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s10.postimg.cc/rdm0kbtsp/google_net_1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="756" data-original-width="756" height="320" src="https://s10.postimg.cc/rdm0kbtsp/google_net_1.png" width="320" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11pt; font-weight: bold;">CDN (Content Delivery Network)</span><span style="font-family: "calibri";"><span style="font-size: 11pt;"> is also worth mentioning. It's enabled by Edge Cache Sites (Edge PoPs, or the light green circule above), the places where the online content can be delivered closer to the users for faster response times. It works with Load Balancing, and the Content is CACHED in 80+ Edge Cache Sites around the globe. </span><span style="font-size: 14.6667px;">unlike most CDNs, <b>your site gets a single IP address that works everywhere</b>, combining global performance with easy management — no regional DNS required.</span></span><span style="font-family: "calibri"; font-size: 11pt;"> For more information <a href="https://cloud.google.com/cdn/">check out the official Google docs.</a></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h1 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
Connecting
your environment to GCP (Cloud Interconnect)</h1>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
While this may
change in the future, a VPN hosted on GCP does not allow for client
connections. However, connecting a VPC to an on-premises VPN (not hosted on
GCP) is not an issue.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There are 3 ways you
can connect your Data Center to GCP:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Cloud
VPN/IPsec VPN</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
as in a standard Site to Site VPN IPsec session (<b>supports IKEv1 and v2</b>).
Supports up to </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">1,5-3 Gbps per tunne</span><span style="font-family: "calibri"; font-size: 11.0pt;">l, but you can set up various to increase
performance. You can also use this option to connect different VPCs to
each other, or your VPC to other Public Cloud. </span><b>Cloud Router</b><span style="font-family: "calibri"; font-size: 11pt;"> </span><span style="font-family: "calibri"; font-size: 11pt;">is not required for Cloud VPN, but it does make things a lot easier, by introducing the Dynamic Routing between your DC and GCP, that supports BGP. When using static routes, any new subnet on the peer end must be added to the tunnel options on the Cloud VPN gateway options.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Dedicated
Interconnect</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
used if you don’t want to go via Internet, and you can meet Google in one
of Dedicated Interconnect points of presence. You would be using Google
Edge Location (you can connect into it Directly, or via Carrier), with
Google Peering Edge (PE) device to which your physical Router (CE)
connects [you need to be in the supported location - Madrid is included].
This is not cheap, currently around <b>1700$ per 10Gbps link, 80GB Max!</b></span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Direct
Peering/Carrier Peering</span><span style="font-family: "calibri"; font-size: 11.0pt;">, which Google does not charge for, but also - <b>there is no SLA</b>.
Peering is a private connection directly into Google Cloud. It's available
in more locations then Dedicated Interconnect, and it can be done directly
with Google (</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Direct Peering</span><span style="font-family: "calibri"; font-size: 11.0pt;">) if you can meet Google's direct peering requirements (Requires you to have a connection in a colocation facility, either
directly or through a carrier provided wave service), or via a Carrier (</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Carrier
Peering</span><span style="font-family: "calibri"; font-size: 11.0pt;">). </span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
And, as always, Google provides a Choice Chart if you're not sure which option is for you:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s10.postimg.cc/nva0nd0g9/google_infra_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="800" data-original-width="751" height="640" src="https://s10.postimg.cc/nva0nd0g9/google_infra_2.png" width="600" /></a></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
</h2>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
How do I transfer my data from my Data Center to GCP?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When transferring
your content into the cloud, you would use the "<b>gsutil</b>" command line
tool, and have in mind:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>Parallel uploads</b> (-o, plus
you need to set the parameters) are <b>for breaking up larger files into
pieces for faster uploads. </b></span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>Multi-threaded uploads (-m)
</b> are for large numbers of smaller files. If you have bunch of small
files, you should group together and compress.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">You can add multiple Cloud
VPNs to reduce the transfer time.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">gsutil by default will by
default occupy the entire bandwidth. There are tools to optimize this.
When it fails, gsutil will retry by default.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">For ongoing automated
transfers, use a </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">cron job</span><span style="font-family: "calibri"; font-size: 11.0pt;">.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<b>Google Transfer
Appliance</b> is a new thing, probably not in the exam, it allows you to copy all
your data, ship it to google, and they will load it to the Cloud for you.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h1 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
Load
Balancing in GCP</h1>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
One of the most
important parts of Google Cloud, because it enables the Elasticity, much needed
in the cloud, by providing the Auto Scaling for the Managed Instance Groups. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Have in mind that
the Load Balancing services for the GCE and GKE work in a different ways, but
basically they achieve the same thing - Auto Scaling. Here is how this works:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">In GCE there is a managed
group of instances generated from the same template (Managed Instance
Group). By enabling a Load Balancing service, you're getting a Global URL
for your Instance Group, that includes the Health check service launched
from the Balancer to the Instances, which is the base trigger of the Auto
Scaling.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">In GKE you'd have a
Kubernetes Cluster, and the entire Elastic operation of your containers is
one of the signature functionalities of the Kubernetes Cluster, so you
don't have to worry about configuring any of this manually. </span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's get deeper
into the types of the Load Balancing (LB) service in GCP. Have in mind that you
should always have in mind the ISO-OSI model, and if you can provide the LB
service on the high level - go for it! This means that if you can do a HTTPS
Balancing, rather go for that then SSL. If you can't go HTTPS - go for SSL. If
your traffic is not encrypted - sure, go for TCP. Only if NONE of this works
for you, you should settle for the simple Network LB Service.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">IMPORTANT</span>: Whenever you are using one of the
encrypted LB Services (HTTPS, SSL/TLS), the Encryption terminates on the Load
Balancer, and then the proper Load Balancer established a separate encrypted
tunnel to each of the Active Instances.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There are 2 types of
Load Balancing on GCP:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">EXTERNAL</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;"> Load Balancing, for an
access from the OUTSIDE (Internet)</span></li>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: bold; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="a">
<li style="font-weight: bold; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">GLOBAL Load Balancing: </span></li>
</ol>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">HTTP/HTTPS Load Balancing</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">SSL Proxy Load Balancing</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">TCP Proxy Load Balancing</span></li>
</ul>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: bold; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="a">
<li style="font-weight: bold; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="2"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">REGIONAL Load Balancing: </span></li>
</ol>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Network Load Balancer
(notice that the Network Load Balancer is NOT Global, only available in a
single region)</span></li>
</ul>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">INTERNAL</span><span style="font-family: "calibri"; font-size: 11.0pt;">, for the inter-tier access
(example - web servers accessing Data Bases)</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-size: 11pt;"> </span></div>
</div>
</div>
</div>
<!--EndFragment-->Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-39607359963114792472018-03-01T08:01:00.000+01:002018-06-15T16:11:12.010+02:00Google Cloud Platform (GCP) - How do I choose among the Storage and Big Data options?<!--StartFragment-->
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Storage options are
extremely important when using a GCP, performance and price wise. I will do a
bit of a non-standard approach for this post. I will first cover the potential
use cases, explain the Hadoop/Standard DB you would use in each case, and then
the GCP option for the same use case. Once that part is done, I will go a bit
deeper into each of GCP Storage and Big Data technologies. This post will
therefore have 2 parts, and an "added value" Anex:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">Which option fits to my use
case?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Technical details on GCP
Storage and Big Data technologies</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Added Value: Object Versioning and Life Cycle management</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h1 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
1. Which
option fits to my use case?</h1>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Before we get into
the use cases, let's make sure we understand the layers of abstraction of
Storage. <span style="font-weight: bold;">Block Storage</span> is a typical
storage carried out by applications, data stored in cylinders, UNSTRUCTURED
DATA WITH NO ABSTRACTION. When you can refer to data using a physical address -
you're using Block Storage. You would normally need some abstraction to use the
storage, it would be rather difficult to reference your data by blocks. <span style="font-weight: bold;">File Storage</span> is a possible abstraction, and it
means you are referring to data using a logical address. In order to do this,
we will need some kind of layer on top of our blocks, an intelligence to make sure that
our blocks underneath are properly organized and stored in the disks, so that
we don't get the corrupt data.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's now focus on
the use cases, and a single question - what kind of data do you need to store?</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://s14.postimg.cc/fmjusv9zl/Screenshot_2018-03-01_07.38.40.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="210" data-original-width="800" height="168" src="https://s14.postimg.cc/fmjusv9zl/Screenshot_2018-03-01_07.38.40.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<!--StartFragment-->
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If you're using
Mobile, the you will be using a slightly different data structures:</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://s14.postimg.cc/f9sgmr4ld/Screenshot_2018-03-01_07.38.47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="141" data-original-width="800" height="70" src="https://s14.postimg.cc/f9sgmr4ld/Screenshot_2018-03-01_07.38.47.png" width="400" /></a></div>
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's now get a bit
deeper into each of the Use Cases, and see what Google Cloud can offer.</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">If you need</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;"> Block
Storage</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">
for your compute VMs/instances, you would obviously be using a Googles IaaS option called
Compute Engine (GCE), and you would create the Disks using:</span></li>
</ol>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<ul>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Persistent disks (Standard or
SSD)</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Local SSD</span></li>
</ul>
</ul>
<ol style="direction: ltr; font-family: Calibri; font-size: 11pt; font-style: normal; margin-bottom: 0in; margin-left: 0.375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="font-weight: normal; margin-bottom: 0px; margin-top: 0px; vertical-align: middle;" value="2"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">If you need to store an
unstructured data, or "Blobs", as Azure calls it, such as Video,
Images and similar Multimedia Files - what you need is a</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;"> Cloud
Storage</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">.
</span></li>
<li style="font-weight: normal; margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">If you need your BI guys to
access your Big Data using an SQL like interface, you'll use a </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">BigQuery</span><span style="font-family: "calibri"; font-size: 11.0pt;">, a Hive-like Google product.
This applies to cases 3 (SQL interface required), and 7 (OLAP/Data Warehouse).</span></li>
<li style="font-weight: normal; margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">To store the NoSQL Documents
like HTML/XML, that have a characteristic pattern, you should use </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">DataStore</span><span style="font-family: "calibri"; font-size: 11.0pt;">. </span></li>
<li style="margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11pt;">For <b>columnar</b> NoSQL data, that
requires fast scanning, use </span><span style="font-family: "calibri"; font-size: 11pt; font-weight: bold;">BigTable</span><span style="font-family: "calibri"; font-size: 11pt; font-weight: normal;"> (GCP equivalent of HBase).</span></li>
<li style="margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11pt; font-weight: normal;">For Transactional Processing,
or OLTP , you should use </span><span style="font-family: "calibri"; font-size: 11pt; font-weight: bold;">Cloud SQL </span><span style="font-family: "calibri"; font-size: 11pt;">(if you prefer open source)</span><span style="font-family: "calibri"; font-size: 11pt; font-weight: bold;"> or Cloud Spanner (</span><span style="font-family: "calibri"; font-size: 11pt;">if you need less latency, and horizontal scaling)</span><span style="font-family: "calibri"; font-size: 11pt; font-weight: normal;">.</span></li>
<li style="font-weight: normal; margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Same like 3.</span></li>
<li style="margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11pt;"><b>Cloud Storage for Firebase</b> is
great for Security when you are doing Mobile.</span></li>
<li style="font-weight: normal; margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Firebase
Realtime DB</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is
great for fast random access with mobile SDK. This is a NoSQL database,
and it remains available even when you're offline.</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h1 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
2. Technical
details on GCP Storage and Big Data technologies</h1>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Storage
- Google Cloud Storage</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Google Cloud Storage
is created in the form of <span style="font-weight: bold;">BUCKETS</span>, that
are globally unique, identified by NAME, more or less like a DNS. <span style="font-size: 14.6667px;">Buckets are STANDALONE, not tied to any Compute or other resources.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<b>TIP</b>: If you want
to use Cloud Storage with a web site, have in mind that you need a Domain
Verification (adding a meta-tag, uploading a special HTML file or directly via
the Search Console).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There are 4 types of
Bucket Storage Classes. You need to be really careful to choose the most
optimal Class for your Use Case, because the ones that are designed not used
frequently are the ones where you'll be charged per access.<span style="mso-spacerun: yes;"> </span><span style="font-weight: bold;">You CAN CHANGE
a Buckets Storage class</span>. The files stored in the Bucket are called
OBJECTS, the <b>Objects can have the Class which is same or "lower" then
the Bucket</b>, and if you change the Bucket storage class - the <span style="font-weight: bold;">Objects will retain their storage class</span>. The
Bucket Storage Classes are:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Multi-regional</span><span style="font-family: "calibri"; font-size: 11.0pt;">, for frequent access from
anywhere around the world. It's used for "Hot Objects", such as
Web Content, it has a </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">99,95% availability</span><span style="font-family: "calibri"; font-size: 11.0pt;">, and it's Geo-redundant.
It's pretty expensive, 0.026/GB/Month.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Regional</span><span style="font-family: "calibri"; font-size: 11.0pt;">, frequent access from one
region, with</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;"> 99,9% availability</span><span style="font-family: "calibri"; font-size: 11.0pt;">, appropriate for storing data used by Cloud Engine
instances. Regilnal class has performance for data intensive computations,
unlike multi-regional.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Nearline</span><span style="font-family: "calibri"; font-size: 11.0pt;"> - access once at month at
max, with </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">99% availability</span><span style="font-family: "calibri"; font-size: 11.0pt;">, costing </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">0.01/GB/month</span><span style="font-family: "calibri"; font-size: 11.0pt;"> with a </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">30 day
minimum duration</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
but it's got ACCESS CHARGES. It can be used for data Backup, DR or
similar.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Coldline</span><span style="font-family: "calibri"; font-size: 11.0pt;"> - access once a year at max,
with same throughput and latency, for </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">0.007/GB/month</span><span style="font-family: "calibri"; font-size: 11.0pt;"> with a </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">90 day
minimum duration</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
so you would be able to retrieve your backup super fast, but you would get
a bit higher bill.. At least your business wouldn’t suffer.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
We can get a data IN
and OUT of Cloud Storage using:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">XML and JSON </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">APIs</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Command Line (</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">gsutil - </span><span style="font-family: "calibri"; font-size: 11.0pt;">a command line tool for
storage manipulation)</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">GSP Console (web)</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Client SDK</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
You can use <span style="font-weight: bold;">TRANSFER SERVICE</span> in order to get your date <b>INTO
the Cloud Storage (not out!)</b>, from AWS S3, http/https, etc. This tool won't let you get
the data out. Basically you would use:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">gsutil</span><span style="font-family: "calibri"; font-size: 11.0pt;"> when copying files for the
first time from on premise.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Transfer
Service</span><span style="font-family: "calibri"; font-size: 11.0pt;"> when
transferring from AWS etc. </span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Cloud Storage is not like Hadoop in the architecture sense, mostly because a HDFS architecture requires a Name Node, which you
need to access A LOT, and this would increase your bill. You can read more about Hadoop and it's Ecosystem <a href="https://matscloud.blogspot.com.es/2018/02/big-data-for-infrastructure-engineers.html">in my previous post, here.</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When you want to
store UNSTRUCTURED data.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Storage
- Cloud SQL and Google Spanner</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
These are both
<b>relational databases</b>, super structured data. Cloud Spanner offers ACID++,
meaning it's perfect for OLTP. It would, however, be too slow and too many
checks for Analytics/BI (OLAP), because OLTP needs strict write consistency,
OLAP does not. Cloud Spanner is Google proprietary, and it offers horizontal
scaling, like bigger data sets.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">*ACID (Atomicity, Consistency, Isolation, Durability)</span>
is a set of properties of database transactions intended to guarantee validity
even in the event of errors, power failures, etc.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
OLTP (Transactional)
Applications.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Storage
- BigTable (Hbase equivalent)</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
BigTable is used for
FAST scanning of <span style="font-weight: bold;">SEQUENTIAL key values with LOW
latency</span> (unlike Datastore, which would be used for non-sequential data).
Bigtable is a columnar database, <span style="font-weight: bold;">good for sparse
data </span>(meaning - missing fields in the table), because similar data is stored next to each other. ACID properties
apply only on the ROW level.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
What is columnar
Data Base? Unlike RDBMS, it is not normalised, and it is <span style="font-weight: bold;">perfect for Sparse data </span>(tables with bunch of
missing values, because the Columns are converted into rows in the Columnar
data store, and the Null value columns are simply not converted. Easy.). Columnar DBs are also great for the <span style="font-weight: bold;">data structures with the Dynamic
Attributes because we can add new columns without changing the schema</span>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Bigtable is
sensitive to hot spotting. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Low Latency,
SEQUENTIAL data.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Storage
- Cloud Datastore (has similarities to MongoDB)</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
This is much simpler
data store then BigTable, similar to MongoDB and CouchDB. It's a key-value
structure, like structured data, <span style="font-weight: bold;">designed to
store documents</span>, and it should not be used for OLTP or OLAP but instead<span style="font-weight: bold;"> for fast lookup on keys</span> (needle in the
haystack type of situation, lookup for non sequential keys).<span style="mso-spacerun: yes;"> </span><span style="font-weight: bold;">Datastore is
similar to RDBMS in that they both use indices for fast lookups</span>. The
difference is that DataStore query execution time depends on the size of
returned result, so it will take the same time if you're querying a dataset of
10 rows or 10.000 rows.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">IMPORTANT</span>: Don’t use DataStore for Write
intensive data, because the indices are <span style="font-weight: bold;">fast to
read, but slow to write</span>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Low Latency,
NON-SEQUENTIAL data (mostly Documents that need to be searched really quickly,
like XML or HTML, that has a characteristic patterns, to which Datastore is
performing <span style="font-weight: bold;">INDEXING</span>). It's <span style="font-weight: bold;">perfect for SCALING of a HIARARCHICAL documents with
Key/Value data</span>. Don't use DataStore if you're using OLTP (Cloud Spanner
is a better. choice) or OLAP/Warehousing (BigQuery is a better choice). Don't
use for unstructured data (Cloud Storage is better here). It's good for Multi
Tenancy (think of HTML, and how the schema can be used to separate data).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Big
Data - Dataproc</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Dataproc is a GCP managed Hadoop + Spark </span>(every
machine in the Cluster includes Hadoop, Hive, Spark and Pig. You need at lease
1 master and 2 workers, and other workers can be Preemptable VMs). Dataproc
uses Google Cloud Storage instead of HDFS, simply because the Hadoop Name Node
would consume a lot of GCE resources.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Dataproc allows you
to move your existing Hadoop to the Cloud seamlessly.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Big
Data - Dataflow</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In charge of
transformation of data, similar to Apache Spark in Hadoop ecosystem. Dataflow
is based on Apache Beam, and it models the flow (PIPELINE) of data and
transforms it as needed. Transform takes one or more Pcollections as input, and
produces an output Pcollection.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Apache Beam uses the
I/O Source and Sink terminology, to represent the original data, and the data
after the transformation. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Whenever you have
one data format on the Source, and you need to deliver it in a different
format, as a Backend you would use something like Apache Spark or Dataflow.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Big
Data - BigQuery</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
BigQuery is not
designed for the low latency use, but it is VERY fast comparing to Hive. It's
not as fast as Bigtable and Datastore which are actually preferred for low
latency. BigQuery is great for OLAP, but it cannot be used for transactional
processing (OLTP).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If you need a Data
Warehouse if your application is OLAP/BA or if you require an SQL interface on
top of Big Data.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Big
Data - Pub/Sub</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Pub/Sub
(Publisher/Subscriber) is a messaging transport system. It can be defined as <span style="font-weight: bold;">messaging Middleware</span>. The subscribers subscribe
to the <span style="font-weight: bold;">TOPIC</span> that the publisher
publishes, after which the Subscriber sends an ACK to the
"Subscription", and the message is deleted from the source. This
message stream is called the <span style="font-weight: bold;">QUEUE</span>.
Message = Data + Attributes (key value pairs). There are two types of
subscribers:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">PUSH
Subscriber</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
where the Apps make HTTPS request to googleapis.com</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">PULL
Subscriber</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
where the Web Hook endpoints able to accept POST requests over HTTPS </span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Perfect for
applications such as Oder Processing, Event Notifications, Logging to multiple
systems, or maybe Streaming data from various Sensors (typical for IoT).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Big
Data - Datalab</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Datalab is an
environment where you can execute notebooks. It's basically a Jupyter or
iPhython for notebooks for running code. Notebooks are better the text files
for Code, because they include Code, Documentation (markdown) and Results.
Notebooks are stored in Google Cloud Storage.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
should I use it?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When you want to use
Notebooks for your code.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Need
some help choosing?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If it's still not
clear which is the best option for you, Google also made a complete Decision
Tree, exactly like in the case of "Compute". </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s14.postimg.cc/jvokuykxt/Storage_Options_GCP.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="450" data-original-width="800" height="360" src="https://s14.postimg.cc/jvokuykxt/Storage_Options_GCP.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<!--StartFragment-->
<!--EndFragment--><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h1 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
3. Added
Value: Object Versioning and Lifecycle Management</h1>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Object
Versioning</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
By default in Google
Cloud Storage If you delete a file in a Bucket, the older file is deleted, and
you can't get it back. When you ENABLE Object Versioning on a Bucket (can only
be enabled per bucket), the previous versions are ARCHIVED, and can be RETRIEVED
later.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When versioning is
enabled, you can perform different actions, for example - use an older file and
override the LIVE version, or similar.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Object
Lifecycle Management</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
To avoid the
archived version creating a chaos in some point of time, it's recommendable to
implement some kind of Lifecycle Management. The previous versions of the file
maintain their own ACL permissions, which may be different then the LIVE one.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Object Lifecycle
Management can turn on the TTL. You can create CONDITIONS or RULES to base your
Object Versioning. This can get much more granular, because you have:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Conditions</span><span style="font-family: "calibri"; font-size: 11.0pt;"> are criteria that must be
met before the action is taken. These are: Object age, Date of Creation,
If it's currently LIVE, Match a Storage Class, and Number of Newer
Versions.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Rules</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Actions</span><span style="font-family: "calibri"; font-size: 11.0pt;">, you can </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">DELETE</span><span style="font-family: "calibri"; font-size: 11.0pt;"> or </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Set another
Storage Class</span><span style="font-family: "calibri"; font-size: 11.0pt;">.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<!--StartFragment-->
<!--EndFragment--></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
This way you can get
pretty imaginative, and for example delete all objects older then 1 year, or
perhaps if a Rule is triggered and conditions are met - change the Class of the
Object from, for example, Regional to Nearline etc.</div>
<br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com1tag:blogger.com,1999:blog-6091645117172561542.post-25731100886328346402018-02-28T20:20:00.000+01:002018-03-04T12:27:21.583+01:00Google Cloud Platform (GCP) - How do I choose among the Compute options? IaaS, PaaS, CaaS/KaaS?<!--StartFragment-->
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Google has made
their Cloud Platform (GCP) so that you can host your application any way your
business requires. When we talk about the traditional Data Center, we tend to
distinguish 3 types of "resources":</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Compute</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Storage</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Networking and Security</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In each of these 3
areas, GCP offers you plenty of options. Don't be naive though, you will need
to know the options quite good in order to optimize your performance and costs.
In this 3-part Blog Post I will go into each of these 3 in detail, and hopefully
help you with your decision.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's start with the
<b>Compute options</b>. There are 3 options to choose from. You can go for the Google App Engine, or a PaaS option, focus on the
code and let Google handle everything else, use a GCP to simply deploy your VMs
(or Instances as they call it) the way you like, or you can choose the Containers.
My idea is to try and explain each of the options in a bit more details. If
this is something you'd be interested in - keep reading.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
What
are IaaS and PaaS?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's start with a
simple question: What are <span style="font-weight: bold;">IaaS (Infrastructure
as a Service)</span>, and <span style="font-weight: bold;">PaaS (Platform as a
Service)</span> and how are they different from a traditional On-premise/Data
Center model? Back to basics - what does our application need in order to run?
Let's start from the bottom of the Application Stack:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Networking</span><span style="font-family: "calibri"; font-size: 11.0pt;">, to be reached, and to reach
data it requires to operate. We need Switches, Routers, etc.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Storage</span><span style="font-family: "calibri"; font-size: 11.0pt;">, to store data. We need
Disks, Storage Cabins, SAN switches.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Servers</span><span style="font-family: "calibri"; font-size: 11.0pt;">, to store the compute loads.
Physical Servers, with RAM, CPUs etc.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Virtualization</span><span style="font-family: "calibri"; font-size: 11.0pt;">, to optimize the usage of
the Physical Resources by using the VMs (Virtual Machines).</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Operating
System</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Middleware</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Runtime</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Data</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Applications</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In the On Premise
architecture, it is on us to (over-) provision and manage all these resources.
Wouldn't it be great if someone would provision and manage some of the
"basic" layers for us, so that we could focus on the part that
actually matters to our business? THIS is what it's all about. I like how AWS
defines this - in IaaS, the Cloud Provider takes care of all the heavy lifting,
or as they call it - <span style="font-weight: bold;">Undifferentiated Services</span>,
while you handle the services on top, that make your business different from
your competitors.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Now, check out the
following diagram, to see what exactly is managed by the Cloud Provider, and
what is managed by You, in the case of IaaS, PaaS and SaaS.</div>
<div style="margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiTuDeja4DfAzhJgtqMz6JzWvOlvdsJZsnCJzfqvdFI3MIjHs6gVuy4-tB68_RuK1yvi3srAPGDcYy9ux_SyTa8_E5iyvgJb6ryksBq7ADYcAi68_kEjTxpP5J-rW3j4jUE2ynxV4i2Wdq/s1600/IaaS.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="410" data-original-width="697" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiTuDeja4DfAzhJgtqMz6JzWvOlvdsJZsnCJzfqvdFI3MIjHs6gVuy4-tB68_RuK1yvi3srAPGDcYy9ux_SyTa8_E5iyvgJb6ryksBq7ADYcAi68_kEjTxpP5J-rW3j4jUE2ynxV4i2Wdq/s400/IaaS.png" width="400" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<i>*In some examples
out there, in IaaS OS is partially managed by You. This pretty much depends on
the model that Cloud Provider is offering.</i></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
What
Compute options does GCP offer?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There are Compute options for hosting your applications in Google Cloud. You can use one of those, or
Mix and Match:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">Google Cloud
Functions</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">
(currently in Beta). It's a Serverless environment for building and
connecting other cloud services. Very simple, very single purpose
functions, written in JavaScript, executed in Node.js. Cloud Function
executes a response to a TRIGGER event. Functions are not exactly a Compute option, but they do match this use case, so I'll just keep them here.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Google App
Engine (GAE)</span><span style="font-family: "calibri"; font-size: 11.0pt;">, is
the PaaS option, serverless and ops free. It's a flexible, zero ops,
serverless platform for highly available apps. You would choose GAE if you
ONLY want to focus on writing code. It can be used for the Web sites,
Mobile apps, or gaming backend, and also for IoT. Google App Engine is a
MANAGED SERVICE, meaning - you NEVER need to worry about the
infrastructure, it's invisible to you. There are 2 available environments:
</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Standard</span><span style="font-family: "calibri"; font-size: 11.0pt;"> (predefined Runtime) and </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Flexible</span><span style="font-family: "calibri"; font-size: 11.0pt;"> (configurable Runtime). We will get into these in more detail.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Google
Kubernetes Engine (GKE)</span><span style="font-family: "calibri"; font-size: 11.0pt;">, is the CaaS/KaaS (Containers/Kubernetes as a Service) option,
clusters of machines running Kubernetes and hosting containers. Containers
are the auto-contained services, containing all the Libraries and
Dependencies, so that you don't have to worry about the Operating System at all.
GKE engine allows you to focus on the Applications, not on the OS. You
should use it to increase velocity and improve operability by separating
the application from the OS. Ideal for Hybrid applications.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Google
Compute Engine (GCE)</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is the IaaS option, fully controllable down to the OS. We're
talking about Instances of VMs. You should use it if you have a very
specific requirements from your operating system, or if you need to use
the GPUs (yes, this is the only option that let's you add Graphical
Processing Units for intensive compute tasks to your Compute resources).</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There is also a
fifth option called Firebase, and it's specific for Mobile, but I won't go into
that right now. Instead, let's focus on each of the four options mentioned
above. Each of these options can be used for any application, and it's on you
to choose the one that best fits, each one has their Pros and Cons. Yes, you
can mix them in the same application! Check out the following diagram, to get a
clearer picture:</div>
<div style="margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDMe6BV8TdTw5Z_LWjFgnEV5jWeANuJ0Bd_g97ni754X4_yygg33LvkLmPmpSsexIayVHk4eh5eMoKycjwoPQ0S_p3gp348QTFRtCqhJrirwDnIul_Jfy74ap4BwtsQNzos4IMhx90-Xf9/s1600/IaaS2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="413" data-original-width="952" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDMe6BV8TdTw5Z_LWjFgnEV5jWeANuJ0Bd_g97ni754X4_yygg33LvkLmPmpSsexIayVHk4eh5eMoKycjwoPQ0S_p3gp348QTFRtCqhJrirwDnIul_Jfy74ap4BwtsQNzos4IMhx90-Xf9/s400/IaaS2.png" width="400" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Google
Cloud Launcher</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Before we get into
more detail about the Compute options, I'd like to cover the Cloud Launcher,
one of my favorite tools in the GCP. Google Cloud Launcher can help you set up
an easy app, such as WordPress or LAMP stack, in a few minutes. You can customize
your application, because you will have full control of your instances. You
will also know more or less how much everything will cost before you deploy it
all. Remember this for now, because I will be mentioning the Launcher later.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Google
Cloud Functions</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Floating, serverless
execution environments, for building and connecting the cloud environments. You
would be writing simple, single-purpose functions. <span style="font-weight: bold;">When an event that is being watched is fired - Cloud Function is
triggered. </span>You can run it in any standard Node.js runtime. This would be
a perfect option for the coders that like to write their applications in
functions.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Google
Application Engine (GAE) - PaaS</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
PaaS option is a
perfect option if you just want to focus on your code, and you trust Google to
manage your entire infrastructure, including the Operating System. It tends to
be very popular with SW, mobile and Web developers. <span style="font-weight: bold;">If you prefer to pay per use, and not per allocation, you might prefer
PaaS (No-Ops) to IaaS (DevOps) option</span>. Also, there's no vendor lock-in,
you can easily move your Apps to another platform because everything is built
on the Open Source tools. App Engine is <b>REAGIONAL</b>, and Google makes sure that
you have the HA using different (availability) zones within the region.<br />
<br />
<!--StartFragment-->
<!--EndFragment--><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Can you use GAE in Multiple Regions?</span> You cannot
change the region. Your app will be served from the region you chose when
creating the app. Anyone can use the app, but users closer to the selected
region will have lower latency. More details: <a href="https://cloud.google.com/appengine/docs/locations">https://cloud.google.com/appengine/docs/locations</a></div>
</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<b>App Engine supports ONLY HTTP/S.</b></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
GAE is super easy to
use. You will basically need to create a new Folder, store your files in there, and execute the
command "<span style="font-weight: bold;">gcloud app deploy</span>".
That's it!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
There are two
environments, depending whether you can customize an OS:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">Standard
(deployed in Containers)</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">, preconfigured with one of the several available
runtimes (specific versions of Java 7, Python 2, Go, PHP). Each runtime
includes the standard Libraries. Basically this is a container -
Serverless. Your code is running in a Sandboxed environment.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Flexible
(deployed in VM Instances, based on GCE)</span><span style="font-family: "calibri"; font-size: 11.0pt;">, that you can customize into a non standard
environment, and you can use Java 8, Python 3.x, .NET, also supporting
Node.js, Ruby, C#. This is not a container, it's a VM of a compute
instance, and </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">you are charged based on the usage of the VM instance
(CPU, memory, disk usage) that's been provisioned for you</span><span style="font-family: "calibri"; font-size: 11.0pt;">. Unlike on GCE, the
instances are automatically managed for you, meaning - regional placement,
updates, patches and all (root SSH disabled by default, but can be
enabled).</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">IMPORTANT</span>: Scale up time is measured in seconds
in Standard environment, and in minutes in the Flexible environment, simply
because the containers are much faster then the VM instances.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Google
Compute Engine (GCE) - IaaS</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Google Compute Engine </span>should be used when you
need IaaS, for example, you need to tune your Load Balancing and Scaling. When
you create a VM instance (<span style="font-weight: bold;">each instance needs to
belong to a Project, and a Project can have up to 5 VPC - Virtual Private
Networks</span>), you need to choose the Machine Type, a Zone, an Operating
System (Linux and Windows Server are available, you get root access and SSH/RDP
enabled). You can choose one of the following Machine Types, but have in mind
that in order to later change it you need to stop the instance, change it, and
then turn it back on:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Standard</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">High memory</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">High CPU</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Shared core (small, non
resource intensive)</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Compute Engine
instances are <b>pay-per-allocation</b>. When the instance is running, it is charged
at an per-second rate whether it is being used or not. I'd also like to use this
section to clarify a few important concepts related to GCE:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">What is a PREEMPTABLE
instance?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">How does Google Maintenance
affect your workloads?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">How do I automate instance
creation?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">What Disks can I assign to my
Instance?</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Which VMs and Images are
available for me, and can I qualify for discounts?</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
What's
a Preemptable VM instance?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
A type of VM
instance that can be deleted with 30 second notification time, once the SOFT
OFF signal is sent (best practice: you need a SHUT DOWN SCRIPT, able to shut the instance off
and do all the clean-up in less then 30 seconds). It's much cheaper, of course,
because it can be deleted <span style="font-weight: bold;">AT ANY TIME (at least
once every 24 hours)</span>. It can, for example, be used for the fault
tolerant applications.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="color: red;"> </span></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
How
does Google Maintenance affect your workloads?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Google can shut down
your machine for maintenance. You can configure what to do in this case,
migrate or terminate. This is your call, as it directly depends on the nature
of your application, and whether they are Cloud Native (instances treated as a Cattle, rather then as Pets. Confused? <a href="https://matscloud.blogspot.com.es/2018/02/google-cloud-architect-why-and-how-to.html">Read my previous post for clarification</a>).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Live Migration</span> allows an instance to be up and
running, even in the maintenance state, or during a HW or SW update, failed HW,
network and power grid maintenance etc. The <b>instance is moved to another host
in the same zone</b>. VM gets a notification that it needs to be evicted. A new VM
is selected for migration, and the connection is AUTHENTICATED between the old
and the new VM.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
When a Live
Migration is executed there are 3 stages:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">Pre-migration
brownout</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">:
VM executing on the source, when most of the state is sent from source to
target. The time depends on the memory that needs to be copied and
similar.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Blackout</span><span style="font-family: "calibri"; font-size: 11.0pt;">: a brief moment when none of
the VMs are running. </span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Post-migration
brownout</span><span style="font-family: "calibri"; font-size: 11.0pt;">: a VM
is running on the destination/Target Host, but the source VM is still not
killed, ready to provide support if needed.</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<b>IMPORTANT</b>:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: 0.375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="color: red; margin-bottom: 0px; margin-top: 0px; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11pt; font-weight: bold;">Preemptable instances cannot be live migrated.</span></li>
</ul>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="color: red; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Live
migration cannot be used for the VMs with GPUs.</span></li>
<li style="color: red; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Instances
with the local SSD can be live migrated. </span></li>
</ul>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
</h3>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
How
do I automate instance creation?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
To <span style="font-weight: bold;">AUTOMATE</span> the instance creation, you can use the
<span style="font-weight: bold;">gcloud command line</span>. One of the options
is for example to assign a <span style="font-weight: bold;">LABEL</span> to
instances you want to group (called <span style="font-weight: bold;">Instance
Group</span>) in order to monitor or automate. You can get the exact script to,
for example, create an instance, from the graphical interface, just look for
the API and command line equivalents. Yes, this is awesome, you can literally
get an API for any graphical interface action you take. Automation made easy,
good job Google!</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
DevOps tools are
also available (GCP equivalents for some), which is great if you have strong
DevOps skills in the house:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Compute Engine Management
using Puppet, Chef, Ansible.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Automated Image Builds with
Jenkins, Packer and Kubernetes.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Distributed Load Testing with
Kubernetes.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Continuous Delivery with
Travis CI.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Managing Deployments using
Spinnaker.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
What
Disks can I assign to my Instance?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
You also have loads
of Storage options for your instances. I won't go into the storage options here in detail, but to create a Disk for your VM instance you have 4 options:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>Cloud Storage Bucket</b>, as the
cheapest option.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>Standard persistent disks</b> (64
TB).</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>SSD persistent disks</b> (64 TB).</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;"><b>Local SSD</b> (3 TB), actually
attached to the instance, in the same Server.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
Which
VMs and Images are available for me, and how do I qualify for discounts?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Images help you
instantiate new VMs with the OS already installed. There are <span style="font-weight: bold;">Standard</span> and <span style="font-weight: bold;">Premium
Images</span>, depending whether you need some kind of license, like for RedHat
Enterprise Linux or MS Windows. You should have in mind that you have 2
possibilities to get your image ready to launch:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Startup
Script</span><span style="font-family: "calibri"; font-size: 11.0pt;">, that you
need to write in order for it to download your dependencies, and prepare
everything. It needs to always bring the VM in the same state, regardless
how many times you execute it.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Baking</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is a more efficient way to
create an image in order to faster provision an instance, much more
efficient then a Startup script. You would start from the Premium image,
and create a Custom instance (sort of a Template, if you will). Baking
takes much shorter to provision an instance then a Startup disk.
Everything is included into the "baking image". Version
management and rollbacks are much easier, you can just rollback an image
as a whole.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Check out <a href="https://cloud.google.com/pricing/">this link about Google Cloud pricing</a> for more details.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In the image
lifecycle the possible statuses are: <span style="font-weight: bold;">CURRENT,
DEPRECATED</span> (can still be used and launched), <span style="font-weight: bold;">OBSOLETE</span> (cannot be launched) and <span style="font-weight: bold;">DELETED</span>
(cannot be used). This should give you some idea about how you would be
managing your instance versions.</div>
<div style="color: red; font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="color: red; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Snapshots
can only be accessed within the same project.</span></li>
<li style="color: red; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">All machines
are charged for at least 1 minute. After that, a per-second payment is
applied. The more you use the VM, the more discount you get.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Before we get to the
possible discounts, you first need to choose your machine type correctly, to
optimize the cost and the performance:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">Pre-defined</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Custom</span><span style="font-family: "calibri"; font-size: 11.0pt;">: You can specify the number
of vCPUs and Memory. You would start with one of the pre-defined, and if
you see that your CPU or memory are under-utilized, customize it.</span></li>
</ol>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: bold; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="font-weight: bold; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="3"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">Shared-core</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;"> is another option, meant for small,
non resource intensive applications, that require BURSTING. </span></li>
</ol>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="4"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">High Memory
Machines</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">:
more memory per vCPU, 6.5GB per Core.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">High CPU
Machines</span><span style="font-family: "calibri"; font-size: 11.0pt;">: more
vCPU per unit of memory, 1.8GB per Core</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Google offers a few
types of discount/price optimization, among others:</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">Sustained
use</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">, when you
use a VM for a long period of time</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Committed
use</span><span style="font-family: "calibri"; font-size: 11.0pt;">, that you
can purchase in 1 year or 3 year contract, and you get a good price.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Rightsizing</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is a feature recommends
which size of the VMs to run after analyzing your application behavior.
This is a brand new feature, and it </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">relies to the Stackdriver collected
information from the past 8 days</span><span style="font-family: "calibri"; font-size: 11.0pt;">.</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Google
Containers/Kubernetes Engine (GKE) - CaaS/KaaS</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If you have lots of
dependencies, you would of course benefit most using the Containers. Container
is a light weight standalone executable package that includes everything needed
to run it: code, runtime, system tools, system libraries, settings. Containers
de-couple the Application from the Operating System, and they can reliably run
on different environments. Different containers run on a same Kernel, as
presented in the picture below, taken from the Dockers web page:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGWhLFmjzLdFydm1BZASA-dH_fvHJEl_-bUMTfMKIu-vG8VwVzGewVPwrCEMJy-D-jhaoBp6D1J4m5aAdTN2z_bA-LHFt3xrt9RDOw-_8umXmzJycnmAncMQz52T1dvMy32zzeD4Q3xWUc/s1600/docker1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1273" data-original-width="1600" height="253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGWhLFmjzLdFydm1BZASA-dH_fvHJEl_-bUMTfMKIu-vG8VwVzGewVPwrCEMJy-D-jhaoBp6D1J4m5aAdTN2z_bA-LHFt3xrt9RDOw-_8umXmzJycnmAncMQz52T1dvMy32zzeD4Q3xWUc/s320/docker1.png" width="320" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
Container
vs VM</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
A VM contains an
entire operating system packaged along with the application. A container ONLY
runs an OS Kernel and nothing else, it contains the Application and the
essential Libraries, Binary files etc., and it can easily be moved from one
Physical or Virtual machine that has the Kubernetes engine, to another.
Containers are much faster, as there is no OS to boot, and they are much
smaller in size.</div>
<div style="margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSdPHf4WQyJE5PjOSxn5cL_tb8PpII8667WmUmpQMx7M-aeSTv3mtJyiF68FpIO3qsGKBXI2rdDUyfKK5RVsSGE1NIFgotL6EkBUIRoH8PN-zxuUruCwV_GKfoohNGZCDjApifyTYwNbj5/s1600/container+vs+vm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="848" data-original-width="1434" height="236" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhSdPHf4WQyJE5PjOSxn5cL_tb8PpII8667WmUmpQMx7M-aeSTv3mtJyiF68FpIO3qsGKBXI2rdDUyfKK5RVsSGE1NIFgotL6EkBUIRoH8PN-zxuUruCwV_GKfoohNGZCDjApifyTYwNbj5/s400/container+vs+vm.png" width="400" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
To be precise, using
Containers/Dockers we can achieve:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Process isolation</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Namespace isolation</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Own Network Interface</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Own Filesystem</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Meanwhile, when we
say a <span style="font-weight: bold;">Micro service, that simply means that one
container = one process</span>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
What
is Kubernetes?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Kubernetes is an
open source Container Manager, originally created by Google for it´s internal
use. Kubernetes automates Deployment, Scaling and Management. This means that
using Kubernetes you can:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Rollout new features
seamlessly</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Auto scale your application</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Run your application in the
Hybrid environment, as long as you have the Kubernetes Engine in your VMs.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Why is Kubernetes so
important here? Because Google Kubernetes Engine uses Kubernetes as a Container
Management engine. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's first check
out the important <span style="font-weight: bold;">components of the Kubernetes
architecture</span>:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">A </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Container
Cluster</span><span style="font-family: "calibri"; font-size: 11.0pt;"> has one
supervising machine running Kubernetes (</span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Master Endpoint</span><span style="font-family: "calibri"; font-size: 11.0pt;">, or </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Master
Instance</span><span style="font-family: "calibri"; font-size: 11.0pt;"> works
like Hadoop Cluster Manager). </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Kubernetes Master manages the
cluster</span><span style="font-family: "calibri"; font-size: 11.0pt;">, and
it's your single point of management of the Kubernetes Cluster.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Master Instance will be in
touch with a number of individual machines using a software called </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Kubelet</span><span style="font-family: "calibri"; font-size: 11.0pt;">, each running Docker.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Each individual machine
running Kubelet is known as a </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Node Instance</span><span style="font-family: "calibri"; font-size: 11.0pt;">.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Pod is a
smallest deployable unit, a group of 1 or more containers in a Node</span><span style="font-family: "calibri"; font-size: 11.0pt;">. Inside each Pod in every
Node Instance, Containers are running. Pod has it's settings in a
Template.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Replication
Controller</span><span style="font-family: "calibri"; font-size: 11.0pt;">
ensures that specific number of Pod replicas are running across Nodes.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Services</span><span style="font-family: "calibri"; font-size: 11.0pt;"> are the abstraction layer
that decouples the frontend clients to the backend pods. They define the
LOGICAL set of pods across nodes and the way of accessing them. Load
Balancing is one of the Services, creating an IP and a port as a
connection point to our Pods.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Label</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is a METADATA with semantic
meaning. It's used for selecting and grouping the objects.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">scheduler</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is in charge of scheduling
pods onto nodes. Basically it works like this: You create a pod,</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
scheduler
notices that the new pod you created doesn’t have a node assigned to it, and
assigns a node to the pod. It’s not responsible for actually running the pod –
that’s the kubelet’s job. So it basically just needs to make sure every pod has
a node assigned to it.</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">kubectl</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is a CLI tool for
Kubernetes.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimRNt_BgeoOexXyP62EDiUqj6Qy60aDXxboNYHYyfecCL0aS4yKT-oYPOVEHX0UUyYlppGNDABVf4qwmKgKCtYF6huRvU_LJJ8KTWa6svQDSbBpmah-W5z0HmW_HfprvxVvcUUEUzWn0DI/s1600/kubernetes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="359" data-original-width="638" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimRNt_BgeoOexXyP62EDiUqj6Qy60aDXxboNYHYyfecCL0aS4yKT-oYPOVEHX0UUyYlppGNDABVf4qwmKgKCtYF6huRvU_LJJ8KTWa6svQDSbBpmah-W5z0HmW_HfprvxVvcUUEUzWn0DI/s400/kubernetes.png" width="400" /></a></div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11pt;"><br /></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri"; font-size: 11pt;">Google Cloud Engine
includes the following components, most clarified in the Kubernetes
architecture:</span></div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Container
Cluster</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
includes a Kubernetes Master and Compute Engine instances where Kubernetes
are running, managing all the components with Kubernetes Master.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Kubernetes
Master</span><span style="font-family: "calibri"; font-size: 11.0pt;">, as a
single point of management of the cluster.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Pods</span><span style="font-family: "calibri"; font-size: 11.0pt;">, as groups of containers.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Nodes</span><span style="font-family: "calibri"; font-size: 11.0pt;">, as individual Compute
Instances.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Replication
Controller</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
ensuring the defined number of Pods are always available.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Services</span><span style="font-family: "calibri"; font-size: 11.0pt;">, decoupling a frontend
client from the backend Pods, providing a Load Balancer with a single URL
to access your Backend.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Container
Registry</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is the
image repository, so that you can deploy container images</span></li>
</ul>
<div>
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div>
<span style="font-family: "calibri";"><span style="font-size: 14.6667px;"><br /></span></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjanLR54CRwTFMlA4iZQF0-nG3wb_LCS7TCzCbLR4T0Of8J0Jp2p8GGVnsMYRSUkf7BwSZ7Q19Fhpo4Sp5OjspE9xOJuY41RpUNF6asUXrV0gjNXurxCLMVPiktY3HgRlvdET3B0Gb4sz7Q/s1600/kub2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="524" data-original-width="1062" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjanLR54CRwTFMlA4iZQF0-nG3wb_LCS7TCzCbLR4T0Of8J0Jp2p8GGVnsMYRSUkf7BwSZ7Q19Fhpo4Sp5OjspE9xOJuY41RpUNF6asUXrV0gjNXurxCLMVPiktY3HgRlvdET3B0Gb4sz7Q/s400/kub2.png" width="400" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
Why
GKE, and not Kubernetes on GCE?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
This all depends on
what exactly are your needs. You can use CaaS by Google (GKE), which is easier
out of the box, and Google would manage the entire "Undifferentiated"
application stack, up to Containers. You can also build your own Container management
on top of Googles IaaS (GCE), for example if you need GPUs, or you have some
specific OS needs, or maybe a non-Kubernetes container solution, or if you are
migrating your existing on premise Container solution. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Before you make a
decision to, for example, run Kubernetes directly without something like GKE on
top of it, I strongly recommend you to investigate the following GitHub link,
on implementing Kubernetes without the pre-defined scripts: <a href="https://github.com/kelseyhightower/kubernetes-the-hard-way" style="font-size: 14.6667px;">https://github.com/kelseyhightower/kubernetes-the-hard-way</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If you use
containers, the best way would be to use DevOps methodology, and Jenkins for
CI/CD. You can use <span style="font-weight: bold;">Stackdriver</span> for
logging and monitoring.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Storage options for
GKE are the same like with the GCE<span style="font-weight: bold;">, </span>but<span style="font-weight: bold;"> Container disks are ephemeral</span> (lasting for a
very short time), so if you do want your data not ephemeral, you need to use an
abstraction called <span style="font-weight: bold;">gcePersistentDisk</span>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #377bac; font-family: Calibri; font-size: 12.0pt; margin: 0in;">
When
would you use GKE instead of GAE?</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
GAE only supports
HTTP/HTTPS, so if you need to use any other protocol - you would go for CaaS
rather then App Engine.<span style="mso-spacerun: yes;"> </span>Also, if you are
using a Multicloud environment, GAE only works on GCP. App Engine doesn't use Kubernetes,
so if you want to use Kubernetes - you would also rather go for GKE.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Interesting fact</span>: Pokemon GO was deployed on
GKE (50x more users connected then expected), while Super Mario Run (launched
at 150 countries at the same time) was deployed on the GAE.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Need
some help choosing?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
If it's still not
clear which is the best option for you, Google also made a complete Decision
Tree. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOlL66XDUCdzIKmVIU0LPNl3uVTgeEhUR4eDZbIIicRhnLyJ-4bctiwy4zZqk2PaK6uYrTYunIYyFN1ww-3XmgJKGRzkO3kCNf1byF4oR8tA_gsWi9PgSQBZRcfU1_GWarCbq0eEWwg3T0/s1600/ChooseCompute.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="628" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOlL66XDUCdzIKmVIU0LPNl3uVTgeEhUR4eDZbIIicRhnLyJ-4bctiwy4zZqk2PaK6uYrTYunIYyFN1ww-3XmgJKGRzkO3kCNf1byF4oR8tA_gsWi9PgSQBZRcfU1_GWarCbq0eEWwg3T0/s640/ChooseCompute.png" width="249" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<!--EndFragment--><br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com3tag:blogger.com,1999:blog-6091645117172561542.post-65914060383915142892018-02-27T19:01:00.002+01:002018-02-27T19:01:43.520+01:00Big Data for Infrastructure Engineers: What is Hadoop?
<!--StartFragment-->
<br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Before we start,
let's make sure we know what we're talking about here. A few concepts need to
be clarified:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">OLTP vs OLAP</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Big Data</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">Machine Learning</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
OLTP
and OLAP</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
We can divide IT
systems into transactional (OLTP) and analytical (OLAP). In general we can
assume that OLTP systems provide source data to data warehouses, whereas OLAP
systems help to analyze it. Taking this into the Data Bases world, we
have:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">OLTP - On Line Transactional Processing</span>,
normally handled by Relational Data Bases</div>
<div style="font-size: 11.0pt; margin: 0in;">
<span style="font-family: Calibri; font-weight: bold;">OLAP - On Line Analytical Processing</span><span style="font-family: Calibri;"> (Business Analytics and advanced data processing),
requires a Big Data technology, like Hive. </span><span style="color: #333333; font-family: q_serif; font-weight: bold;">Business Intelligence</span><span style="font-family: Calibri;"> (OLAP) refers to the generation of reports
which may or may not involve sophisticated tools like Cognos or Business
Objects.</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Big
Data</h3>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Big Data tends to
refer to the extremely large data sets that may be analysed computationally to
reveal patterns, trends, and associations, especially relating to human
behavior and interactions.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h3 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
Machine
Learning</h3>
<div style="font-size: 11.0pt; margin: 0in;">
<span style="color: #333333; font-family: q_serif; font-weight: bold;">Machine Learning</span><span style="font-family: Calibri;">, or as I would call it BI 2.0, does exactly as its
name suggests. Enabling the Machine to learn from the data. Any forward looking
activity, i.e. whenever the term “Predictive” kicks in, you can expect Machine
Leaning to be there. </span><span style="color: #333333; font-family: q_serif;">At
the same time, it can also be used for Business intelligence. Example: predict
orders next week, identify a fraudulent insurance claim, power a chat-bots to
provide L1 support to customers and so on. Machine learning has wider
applications. It can be leveraged to power businesses as well. And since
business generates lot of data, it makes a easy ground for machine
learning. The idea is always the same:</span></div>
<div style="color: #333333; font-family: q_serif; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="color: #333333; font-family: q_serif; font-size: 11.0pt; margin: 0in;">
How
does the Machine Learning work? Simply get a bunch of Data, proceed to the
"<span style="font-weight: bold;">Training</span>" analyzing the data,
and get the conclusions (<span style="font-weight: bold;">Model</span>) in order
to later be able to make <span style="font-weight: bold;">Predictions</span>,
applicable to the data that hasn't been used for training.</div>
<div style="color: #333333; font-family: q_serif; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="color: #333333; font-family: q_serif; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">DATA -> TRAINING -> MODEL -> PREDICTIONS</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Have in mind that
Model is a "living" thing, as the new data is continuously being
brought in, the Model is being continuously improved.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #2e75b5; font-family: Calibri; font-size: 14.0pt; margin: 0in;">
What
is Hadoop?</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Relational Data
Bases or RDBMS (Relational Data Base Management System) are not optimized for
the needs of Big Data and Machine Learning. We need a File System, that permits
us to scale horizontally, and allows us to perform BA. Enter - Hadoop. Hadoop
is based on the Distributed Computing principles, meaning - lots of cheap
hardware, and prepared for horizontal scaling, unlike Monolithic Computing,
where you'd rely on a single Super Computer. To get the naming right, remember
that <span style="font-weight: bold;">Hadoop Clusters are composed of Nodes, that
run in Server Farms</span>.</div>
<div style="color: #333333; font-family: q_serif; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="color: #333333; font-family: q_serif; font-size: 11.0pt; margin: 0in;">
Simply
put,<span style="font-weight: bold;"> Hadoop = HDFS + MapReduce (+ YARN)</span>.
Let's now demystify this… We have 3 components:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="color: #333333; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: q_serif; font-size: 11.0pt; font-weight: bold;">HDFS</span><span style="font-family: q_serif; font-size: 11.0pt;">, or the Hadoop File System</span></li>
<li style="color: #333333; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: q_serif; font-size: 11.0pt; font-weight: bold;">MapReduce</span><span style="font-family: q_serif; font-size: 11.0pt;">, for data representation,
using Java</span></li>
<li style="color: #333333; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: q_serif; font-size: 11.0pt; font-weight: bold;">YARN</span><span style="font-family: q_serif; font-size: 11.0pt;">, in charge of Replication
and Clustering</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">HDFS</span>: Lots of cheap hardware where the
Distributed Computing is "stored". Google File System was created to
solve the Distributed Storage problem, and Apache developed the Open Source
version of this called HDFS. It's not optimized for low latency, but it has an
"insane" throughput. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In HDFS we will have
a bunch of cheap servers, with the Hard Drives. Each of these will be known as
a Node, and there will be one Master Node, called the <span style="font-weight: bold;">Name Node</span>, containing the Metadata for all the other Nodes, known
as the Data Nodes. Name Node knows where the stuff is, but the Data Nodes
contain the Data.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
HDFS works as a
Block Storage, because the large files are separated in many same sized (128MB)
Blocks, and stored in different Data Nodes. Each node contains a partition of a
split of data. Name node has the "Table of Contents" where different
Block locations are documented. For example, Block 7 is in DN4, as in the
diagram below.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAPWL2_fJ1905S4UpyNZl9rjQ0hAthpWsMezzJ6ZXqfnckWObCiiHz2f3SqnAq4s55TQBiB9e0Y85_E0UuO5s-nDOTh2UwR2UepZ0ShUBj8a06vTUJy0L12iwgjCuuabo7p3NzI3SGg_yp/s1600/Block+Storage.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="686" data-original-width="1417" height="154" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAPWL2_fJ1905S4UpyNZl9rjQ0hAthpWsMezzJ6ZXqfnckWObCiiHz2f3SqnAq4s55TQBiB9e0Y85_E0UuO5s-nDOTh2UwR2UepZ0ShUBj8a06vTUJy0L12iwgjCuuabo7p3NzI3SGg_yp/s320/Block+Storage.png" width="320" /></a></div>
<div style="margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
How is HA handled?
Using the <span style="font-weight: bold;">replication factor</span>, meaning
that every Block is stored in more then one Data Note. Name node needs to keep
track of these. A Replication Strategy handles that the Replicas are stored in
an optimal way, to optimize Bandwidth.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Since I'm currently
preparing for the Google Cloud Architect exam, I've been investigating how
Hadoop as a Managed Service is handled on the GCP (Google Cloud Platform).
Dataproc is Google's managed Hadoop, which let's you not worry about the
Replication or Name Nodes. Google Cloud Storage is used on the GCP instead of
the HDFS, because if you followed the model with the Name Node, such as Hadoop,
the Instance (VM) with the Name Node would be spending an insane amounts of
resources. In Google Cloud Storage this is optimized without the Name Node (not
going into details here).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">YARN (Yet Another Resource Negotiator)</span>: a
Resource allocator that lets us do Replication and Fault Tolerance. YARN
coordinates the cluster, using two components:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt; font-weight: bold;">Resource
Manager</span><span style="font-family: Calibri; font-size: 11.0pt;">, on a
master node</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt; font-weight: bold;">Node Manager</span><span style="font-family: Calibri; font-size: 11.0pt;">, running on all other nodes.
This is actually a container, isolated from everything else on the Node.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin-left: .375in; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">MapReduce</span>: Abstraction that allows any
programmer to present the data in the form of Map and Reduce jobs, and enable
the Distributed Computing. The role of MapReduce is to handle a huge amounts of
data. It takes advantage of parallelism. Every step is done in two functions.</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: Calibri; font-family: Calibri; font-size: 11.0pt; font-size: 11.0pt; font-style: normal; font-weight: bold; font-weight: bold;">Map
operations</span><span style="font-family: Calibri; font-family: Calibri; font-size: 11.0pt; font-size: 11.0pt; font-style: normal; font-weight: normal;">:
Express what the body needs to accomplish. Runs in parallel on many of the
machines in the cluster.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt; font-weight: bold;">Reduce
operations</span><span style="font-family: Calibri; font-size: 11.0pt;">:
Distribute all the results of Map operation, and create a final output,
storing it into all the Data nodes, and their execution will happen in
parallel.</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
This is all (map and
reduce operations) written in JAVA, and Business Analysts don’t do JAVA. This
is why an SQL interface, provided by Hive (or by BigQuery, in GCP) is so
important and popular.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
How does Hadoop
work?</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: normal; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: Calibri; font-family: Calibri; font-size: 11.0pt; font-size: 11.0pt; font-style: normal; font-weight: normal;">User defines the Map and
Reduce tasks using the MapReduce API</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">A Job is Triggered, MapReduce
communicates it to YARN </span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: Calibri; font-size: 11.0pt;">YARN decides the Resource
allocation model, and communicates it to HDFS.</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<h2 style="color: #1e4e79; font-family: Calibri; font-size: 16.0pt; margin: 0in;">
Hadoop
Ecosystem</h2>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Hive</span> (along with Spark), which is basically the
same as Google BigQuery on GCP, provides an SQL interface to Hadoop. BigQuery
uses a columnar format called <span style="font-weight: bold;">Capacitor</span>.
Hive is great for High Latency applications (BigQuery doesn't have as high
latency as Hive, it can even be used for almost real-time applications).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Hive runs on top of
Hadoop, and stores it's data in HDFS. HiveQL is an SQL type language, familiar
to analysts and engineers. SQL is optimized for Relational DBs, and HiveQL for
Hive. Hive will TRANSLATE the queries written in SQL in HiveQL into MapReduce.
A Hive user sees data as if it were stored in Tables.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Comparing to
Relational DB, Hive is meant to be used for LARGE datasets (Giga or petabytes),
Read operations to analyze the historical behavior, with Parallel computations
(need more space - add more servers, in accordance with Horizontal Scaling
philosophy) enabled by MapReduce (relational DB runs against one really
powerful server), and remember that <span style="font-weight: bold;">Hive is
designed for the High Latency use</span>, mostly for Read operations.
Relational DB was designed for Low Latency, quick SQL consults, Read and Write
operations.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Hive uses a so
called Bucketing segmentation. Partitioning is designed for a non equal data
segments. Bucketing is designed to evenly distribute data. Since <span style="font-weight: bold;">in HDFS the Blocks are 128MB each</span>, which is why
this concepts fits the Bucketing perfectly.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Hbase</span>, which maps directly to <span style="font-weight: bold;">Google BigTable</span> provides a management system on
top of Hadoop. It integrates with the Application much like a traditional
database. Hbase and Bigtable are columnar data bases, and they are designed for
the low latency use. </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Pig</span> - A data manipulation language. It
transforms the unstructured data into a structured format. You can query this
structured data using Hive. Included in <span style="font-weight: bold;">Google
DataFlow</span>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Spark</span> - A distributed computing engine used
along with Hadoop. Spark acts as an interactive Shell to quickly process
Datasheets. It completely abstracts away the MapReduce complexity in data
transformation. <span style="font-weight: bold;">You can use Spark if you want to
use Scala or Python to operate HDFS and YARN</span>. Spark has a bunch of built
in Data Libraries used for Machine Learning, stream processing, graph
processing etc. Included in <span style="font-weight: bold;">Google DataFlow</span>.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Kafka</span> - Stream processing for unbounded
datasets. Kafka takes streaming data from sources and distributes to sinks.
Google Cloud used a Google Pub/Sub instead of Kafka.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Oozie</span> - a workflow scheduling tool on Hadoop.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
I hope this helps
understand the basics of Hadoop ecosystem and Big Data. Stay tuned for more
posts on how GCP is handling Big Data.</div>
<!--EndFragment--><br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com3tag:blogger.com,1999:blog-6091645117172561542.post-35254651935403264522018-02-20T16:18:00.000+01:002018-11-20T12:19:34.902+01:00Google Cloud Architect, why and how to prepare for the exam<h3>
Why Public Cloud?</h3>
In the last many years I've been strongly focusing on the Cloud. I have to admit though that due to various conversations with my customers, Public and Hybrid models have been getting more and more attention, and I can see how when I hear the problems of their businesses - Public or Hybrid Cloud is the answer! So, why aren't they all moving their services and applications to the cloud yet? 3 reasons actually:<br />
<br />
<ol>
<li>Their applications are not Cloud Native. No, you can't just migrate your VMs from VMware to the Public Cloud and say that you're running a Cloud Architecture, you need to kill your pets and grow your cattle (google "Pets vs Cattle" if this sounds like a complete nonsense, and then check out the diagram below).</li>
<li>They have prejudice about Security and Speed of the Cloud. This is partly true... but Public Clouds are continuously improving their infrastructure. Google has invested a LOT of money in Security and High Speed Networking (global mesh redundant FO network). Most companies cant match this.</li>
<li>They're a bit scared of the horror stories of insanely large bills that some of the pioneers got when they moved to cloud. This just means that you need to be well informed what and how you want to use, you need to plan your application migrations, and have a managed service capable of optimising your costs, performance and price wise.</li>
</ol>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s10.postimg.cc/pzpoznqah/pets_cattle.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="450" data-original-width="800" height="225" src="https://s10.postimg.cc/pzpoznqah/pets_cattle.png" width="400" /></a></div>
<h3>
Why Google Cloud Platform (GCP)?</h3>
With all this in mind, a few months ago I started my path to get certified in Google Cloud Platform as a Cloud Architect. Yes, I know that AWS and Azure are the current market leaders, but let me tell you why my money is on GCP:<br />
<ul>
<li>Google's Cloud is the clear winner when it comes to compute and storage costs.</li>
<li>GCP provides a better approach to discounted long-term usage: Instead of requiring users to reserve instances for long periods of time as AWS does, GCP will automatically provide discounts the longer you use the instance - no reservations required ahead of time. </li>
<li>With 5 years head-start, AWS offers a lot more cloud products and options. Even though GCP doesn't offer as many services (at the moment), I have a strong feeling that it's a sleeping giant, and I want to be there when it wakes up!</li>
<li>Google is also pioneering efficiency, 100% carbon neutral since 2007, 50% less energy use then the typical DC.</li>
<li>Last, but not the least, Azure is tied to the MS customers, AWS has a somewhat closed system of it's own (yes, I know they started using Kubernetes, after a long period of resisting the market), and GCP gives the feeling of a most opened platform, which is extremely important when it comes to the Hybrid IT.</li>
</ul>
<h3>
</h3>
<h3>
How did I prepare for the exam?</h3>
CURRENT STATUS: PASSED!!! Find the details <a href="https://matscloud.blogspot.com.es/2018/03/how-i-passed-google-certified.html">here.</a><br />
<div>
<br /></div>
<h4>
Step 1: Get skilled</h4>
<div>
I passed quite a few difficult exams during my professional career, including Cisco's CCIE and VMware VCIX, so I kinda know what I should be expecting. Google Cloud Architect is a relatively new exam, so there is not enough feedback on the courses that I've found online. I did a few GCP courses, just to get into the "rumbo", and let me tell you what I've found:</div>
<div>
<br /></div>
<div>
<b>Blueprint</b>: First I went to Google's official page, and scheduled, read a blueprint, all the documentation, and got an idea how the exam is. You can read it all <a href="https://cloud.google.com/certification/cloud-architect">here.</a></div>
<div>
<br /></div>
<div>
Then I simply went ahead and scheduled an exam 3 months ahead of the starting date, to give my self motivation to work harder on it.</div>
<div>
<br /></div>
<div>
<b>GCP Account</b>: I opened an account at GCP, and started playing around. Google gives you 300$ to just "enjoy the experience" for 12 months. Don't be lazy, I was amazed with how smoothly it works. There are projects in BETA, some are pretty mature, check out the Compute, Container and App Engine, see what the Google Launcher can do (use this one!).</div>
<div>
<br /></div>
<div>
<b>Udemy</b>: Google's official documentation is complete, but insufficient. I went to Udemy, and purchased the 26 hours and 60 Demos "GCP: Complete Google Data Engineer and Cloud Architect Guide", that you can find <a href="https://www.udemy.com/gcp-data-engineer-and-cloud-architect/">here.</a> This cost me around 20 euros. The Data Engineer part was also important for me, as GCP relies heavily on Data Bases, and I must admit - having spent most of my career in a Data Center with the infrastructure, I'm not really a Data Base expert, so I had a lot to learn. This course was Ok, not too long, and even though it's not the best quality - I'd recommend it as a starting point.</div>
<div>
<br /></div>
<div>
<b>Google Cloud Next '17</b>: Go to the YouTube channel, and just watch all the videos from Google Cloud Next that sound interesting. There's a bunch of great material there.<br />
<br />
<b>Linux Academy</b>: These guys have a really high quality courses, and I strongly recommend you to purchase 1 or 2 months of subscription (49$ a month), depending on how much time you can dedicate to studying weekly, and get all 3 courses done. I did it, and I think once I did, I really got the feel of what GCP is all about:<br />
<ul>
<li>Google Cloud Platform Architect, Part 1</li>
<li>Google Cloud Platform Architect, Part 2</li>
<li>Google Cloud Platform Architect, Part 3</li>
</ul>
</div>
<div>
<br /></div>
<div>
<b>Read</b>: There are so many Blogs, Videos and Materials out there about GCP. Google it, read it, have your opinion, read the comments, that way you'll get the feeling of how the platform has been evolving.</div>
<div>
<br /></div>
<div>
<b>Reflect</b>: If you check out the below diagram, taken from one of the better <a href="https://medium.com/@earlg3/google-cloud-architect-exam-study-materials-5ab327b62bc8">blogs out there,</a> and you're 100% clear about all the flows, you're good to procede with the Step 2.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s10.postimg.cc/g68hnd3qx/GCP_Diagram.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="450" data-original-width="800" height="360" src="https://s10.postimg.cc/g68hnd3qx/GCP_Diagram.png" width="640" /></a></div>
<div>
<br />
<h4>
Step 2: Tell people you're preparing for the Exam</h4>
You can Tweet, Blog or just comment it to few of your colleagues. Having in mind that they'll ask you how the exam went will make you take it more seriously, because in the world of preparing for the exams, the excuses are the low hanging fruits.<br />
<br /></div>
<h4>
Step 3: Build something</h4>
<div>
Even though the exam is theoretical, you will be given actual use cases, so you do need a hands-on experience. I went and started building a Wordpress project for my newest personal site - a Mat's Cloud Wiki-like Hybrid Cloud knowledge base. I'll publish it as soon as I get something nice.</div>
<div>
<br /></div>
<h4>
<b>Step 4: Practice</b></h4>
<div>
Do a MOCK lab, do bunch of practice labs that you can find online (for example, <a href="https://cloud.google.com/certification/practice-exam/cloud-architect">check this out</a>), be sure you understand all the technologies.</div>
<div>
<br /></div>
<h4>
Step 5: Pass the exam!</h4>
<div>
<br /></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com2tag:blogger.com,1999:blog-6091645117172561542.post-69483331550225662122018-02-06T09:19:00.003+01:002018-02-06T09:26:45.131+01:00Why is Hybrid Cloud so important? Cisco Cloud BlogAs you probably know, CiscoLive 2018 was celebrated in Barcelona in January 2018, and I had the pleasure of speaking at 3 sessions, two about ACI Anywhere and Hybrid Cloud and one about IoT. My focus was on how Cisco products can help you with these, products such as Cisco ACI (and ACI Anywhere), Cloud Center, Intersight, Tetration etc. The session recordings are not public yet, but as soon as they are I will be publishing them here.<br />
<br />
In the meantime I'm glad to announce that I've been officially registered as Blogger on Cisco Blog, and I'm giving you the link to my post on the topic of why Hybrid Cloud is so important:<br />
<br />
<a href="https://blogs.cisco.com/cloud/why-our-partnership-with-cisco-is-crucial-on-the-road-to-cloud">https://blogs.cisco.com/cloud/why-our-partnership-with-cisco-is-crucial-on-the-road-to-cloud</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://alln-extcloud-storage.cisco.com/ciscoblogs/5a61fcad6f9c6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="450" data-original-width="800" height="225" src="https://alln-extcloud-storage.cisco.com/ciscoblogs/5a61fcad6f9c6.png" width="400" /></a></div>
<br />
<br />
I hope you like it.Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-47054854393497383032018-01-19T14:52:00.000+01:002018-06-15T16:16:29.833+02:00What is NSX-T, and how to build a Home LabLet me start by saying that I've been a big NSX fan ever since it came to the market. I was one of the first CCIEs to get to the VCIX-NV (VMware Certified Network Virtualization Expert). It provides a single point of management of your Network and Security within the entire VMware environment, and it supports APIs, enabling us to finally include Networks into the DevOps philosophy. Olé! Right? Check out the <a href="https://matscloud.blogspot.com.es/p/blogmap.html" target="_blank">Blog Map</a> for more of my NSX posts.<br />
<br />
While NSX is an awesome product, there are 3 things that I haven't liked, from the bare beginning of the product:<br />
<br />
<ol>
<li><b>Management</b>: You can only manage NSX from your vSphere Web Client. Not great UX, a Web Client Plugin that VMware never really got a decent level… No wonder most of VMware admins prefer the vSphere Thick client (not an option for Network and Security admins).</li>
<li><b>NSX only supports ESXi </b>as a hypervisor. It's the best one, I'll admit, but todays Cloud requires MORE flexibility, not less…</li>
<li><b>No way of managing the Physical Network (Underlay).</b> Regardless of the Overlay Networking philosophy, and the fact that NSX would work perfectly over most low latency high throughput Data Center Fabrics, I've seen cases where this was a deal breaker.</li>
</ol>
<br />
<br />
I'm happy to say that VMware actually addressed 2 of my 3 complains with NSX-T:<br />
<br />
<ul>
<li>You can now manage NSX using a separate GUI interface, all HTML5!!! Finally!</li>
<li>NSX-T supports KVM, and will potentially support bunch of other integrations… and yeah, CONTAINERS!!!</li>
</ul>
<br />
<br />
No wonder I wanted to try it as soon as it was available for download. So I did… and wow, was that a pleasant surprise!!! The installation process is just so clean and nice, I was truly impressed. In this post I'll just cover the Manager and Control Cluster part, and in my next post I'll proceed with the Hypervisor integration and the Data Plane.<br />
<br />
Before I proceed with the deep technical stuff, <b>here are a few facts to help you fall in love with NSX-T</b>:<br />
<br />
<ul>
<li>It's unofficially called "Transformers" because different releases have different Transformers names.</li>
<li>The NSX license is unique. This means that you can buy the license, and use it in NSX-V (vSphere integrated version) or NSX-T.</li>
<li>It supports Containers!!! Yes, I'm aware I always end this with 3 exclamations.</li>
<li>NSX-T is too cool for VxLAN, so it's using a new, and cooler GENEVE* protocol.</li>
</ul>
<br />
<br />
*VXLAN and NVGRE headers both include a 24-bit field. STT specifies a 64-bit field. None requires modifying or replacing existing switches and routers, although some equipment vendors have developed hardware assists to increase the efficiency of one or more of the solutions. Now a new network virtualization standard has emerged -- Generic Network Virtualization Encapsulation (GENEVE) -- that promises to address the perceived limitations of the earlier specifications and support all of the capabilities of VXLAN, NVGRE and STT. Also… very few actually use the "basic" VXLAN implementation, most vendors, like Cisco and VMware use the "on steroid" improved versions, which makes them mutually non compatible. Many believe GENEVE could eventually replace these earlier formats entirely. More details <a href="https://blog.russellbryant.net/2017/05/30/ovn-geneve-vs-vxlan-does-it-matter/" target="_blank">here</a>.<br />
<br />
<br />
<h2>
NSX-T Installation</h2>
<h3>
Step 1: Install a NSX Manager and 3 NSX Controllers as OVA files to vCenter or KVM. You will need 4 mutually routed IP addresses.</h3>
If you need instructions on how to deploy an OVA in your vCenter, you're at the wrong blog :)<br />
<h3>
Step 2: SSH into NSX Manager and all of the Controllers, and do a manual Joining process:</h3>
<h4>
NSX Manager</h4>
<span style="font-family: "courier new" , "courier" , monospace;">NSX CLI (Manager 2.1.0.0.0.7395503). Press ? for command list or enter: help</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-manager></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-manager> get certificate api thumbprint</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">3619a99a934ddacf06792ea0cf0566c1d49223ac5117a73d95c31e5c482ef868</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-manager></span><br />
<br />
<br />
<h4>
Controllers </h4>
(repeat this on all 3 controllers, regardless that you dont have the Control Cluster built):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">NSX CLI (Controller 2.1.0.0.0.7395493). Press ? for command list or enter: help</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> join management-plane NSX-Manager username admin thumbprint 3619a99a934ddacf06792ea0cf0566c1d49223ac5117a73d95c31e5c482ef868</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">% Invalid value for argument <ip-address port="">: NSX-Manager</ip-address></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><ip-address port="">: IP address of NSX manager with optional port.</ip-address></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> join management-plane 10.20.41.120:443 username admin thumbprint 3619a99a934ddacf06792ea0cf0566c1d49223ac5117a73d95c31e5c482ef868</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Password for API user:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node successfully registered and controller restarted</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> get managers</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.120 Connected</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<br />
<h4>
Verify on the NSX Manager:</h4>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-manager> get management-cluster status</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Number of nodes in management cluster: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.120 (UUID 4207150A-9BB0-6283-181E-AAB3924321BF) Online</span><br />
<br />
Management cluster status: <b><span style="color: lime;">STABLE</span></b><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Number of nodes in control cluster: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.121 (UUID cf4e1847-508e-4a73-96df-c2311f96a55b)</span><br />
<br />
Control cluster status: <b><span style="color: red;">UNSTABLE</span></b><br />
<br />
<br />
<h4>
Now other 2 Controllers:</h4>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-2> join management-plane 10.20.41.120:443 username admin thumbprint 3619a99a934ddacf06792ea0cf0566c1d49223ac5117a73d95c31e5c482ef868</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Password for API user:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node successfully registered and controller restarted</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-3> join management-plane 10.20.41.120:443 username admin thumbprint 3619a99a934ddacf06792ea0cf0566c1d49223ac5117a73d95c31e5c482ef868</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Password for API user:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node successfully registered and controller restarted</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-manager> get management-cluster status</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Number of nodes in management cluster: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.120 (UUID 4207150A-9BB0-6283-181E-AAB3924321BF) Online</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Management cluster status: STABLE</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Number of nodes in control cluster: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.121 (UUID cf4e1847-508e-4a73-96df-c2311f96a55b)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.122 (UUID 4517abc2-5244-4327-a3c8-739ac18b0fd7)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.123 (UUID 22ab7f0a-02a4-42b0-93b0-f8cce56c4200)</span><br />
<br />
Control cluster status: <b><span style="color: red;">UNSTABLE</span></b><br />
<br />
Note that the Control Cluster is still UNSTABLE. Let´s fix this…<br />
<br />
<br />
<h3>
Step 3: Build the control cluster:</h3>
After installing the first NSX Controller in your NSX-T deployment, you can initialise the control cluster. Initialising the control cluster is required even if you are setting up a small proof-of-concept environment with only one controller node. If you do not initialise the control cluster, the controller is not able to communicate with the hypervisor hosts.<br />
<br />
Lets first make a Controller Cluster, while we only have 1 controller:<br />
set control-cluster security-model shared-secret<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> set control-cluster security-model shared-secret</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Secret:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Security secret successfully set on the node.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> initialize control-cluster</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Control cluster initialization successful.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> get control-cluster status verbose</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">NSX Controller Status:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">uuid: cf4e1847-508e-4a73-96df-c2311f96a55b</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">is master: false</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">in majority: false</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">This node has not yet joined the cluster.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Cluster Management Server Status:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">uuid rpc address rpc port global id vpn address status</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">f0c3fc47-b07c-4db2-9b9d-f4df7ad2aa62 10.20.41.121 7777 1 169.254.1.1 connected</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Zookeeper Ensemble Status:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Zookeeper Server IP: 169.254.1.1, reachable, ok</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Zookeeper version: 3.5.1-alpha--1, built on 12/16/2017 13:13 GMT</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Latency min/avg/max: 0/0/14</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Received: 212</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Sent: 228</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Connections: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Outstanding: 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Zxid: 0x10000001d</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Mode: leader</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node count: 23</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Connections: /169.254.1.1:40606[1](queued=0,recved=204,sent=221,sid=0x1000034b54f0001,lop=GETD,est=1514475848954,to=40000,lcxid=0xc9,lzxid=0x10000001d,lresp=3485930,llat=0,minlat=0,avglat=0,maxlat=8)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> /169.254.1.1:40652[0](queued=0,recved=1,sent=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<br />
<br />
<br />
To make this "STABLE", create a control CLUSTER, and associate the other Controllers to the NSX Managers. Once you do:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-2> set control-cluster security-model shared-secret secret SHARED_SECRET_YOU_SET_ON_CONTROLLER_1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Security secret successfully set on the node.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-2> get control-cluster certificate thumbprint</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ef25c0c7195907387874ff83c0ce0b1775c9a190c2b27c82f9ad1db3da279c3d</span><br />
<br />
<br />
Once you have those, go to the MASTER Controller, and join the other Controllers to the Cluster:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> join control-cluster 10.20.41.122 thumbprint ef25c0c7195907387874ff83c0ce0b1775c9a190c2b27c82f9ad1db3da279c3d</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 10.20.41.122 has successfully joined the control cluster. Please run 'activate control-cluster' command on the new node.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> join control-cluster 10.20.41.123 thumbprint f192c6e8f33b5c9566db6d5bcd5a305ca7ab2805c36b2bd940e1866e43039274</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Node 10.20.41.123 has successfully joined the control cluster. Please run 'activate control-cluster' command on the new node.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<br />
<br />
Now go back to the controllers 2 and 3, and activate the cluster:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-2> activate control-cluster</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Control cluster activation successful.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1> get control-cluster status</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">uuid: cf4e1847-508e-4a73-96df-c2311f96a55b</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">is master: false</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">in majority: true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">uuid address status</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">cf4e1847-508e-4a73-96df-c2311f96a55b 10.20.41.121 active</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">4517abc2-5244-4327-a3c8-739ac18b0fd7 10.20.41.122 active</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">22ab7f0a-02a4-42b0-93b0-f8cce56c4200 10.20.41.123 active</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-controller-1></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">nsx-t-manager> get management-cluster status</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Number of nodes in management cluster: 1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.120 (UUID 4207150A-9BB0-6283-181E-AAB3924321BF) Online</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Management cluster status: STABLE</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Number of nodes in control cluster: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.121 (UUID cf4e1847-508e-4a73-96df-c2311f96a55b)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.122 (UUID 4517abc2-5244-4327-a3c8-739ac18b0fd7)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- 10.20.41.123 (UUID 22ab7f0a-02a4-42b0-93b0-f8cce56c4200)</span><br />
<br />
Control cluster status: <b><span style="color: lime;">STABLE</span></b><br />
<div>
<br /></div>
<div>
Let's now check out the GUI (you'll LOVE the NSX-T graphical interface btw):</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s10.postimg.cc/abxg4whuh/NSX-_T_screenshot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="362" data-original-width="800" height="181" src="https://s10.postimg.cc/abxg4whuh/NSX-_T_screenshot.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
That's it!!! Now let's go integrate it with the Hypervisors...</div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com1tag:blogger.com,1999:blog-6091645117172561542.post-39392456205582732272018-01-16T17:07:00.002+01:002018-01-19T14:58:14.699+01:00ACI 2018: ACI Anywhere and AVE (Application Virtual Edge)It's 2018, and looking back at 2017 I must say that I'm quite happy, because having all in mind - it seems that Cisco is taking Cloud seriously. Four arguments to prove this:<br />
<br />
<ul>
<li>Intent Based approach</li>
<li>UCS and HyperFlex evolution using Cisco Intersight</li>
<li>Cisco CloudCenter evolving</li>
<li>ACI Anywhere</li>
</ul>
<br />
Let's try and explain all these concepts...<br />
<br />
<h3>
What is Intent Based Data Center?</h3>
Apps are the new business, as our customers customer is the business developer. Intent based networking is really about building networking in accordance to that, follow the intent of the developers in the networking space. This means that what we used to call Policy, we should actually call Intent.<br />
<br />
<h3>
What is Cisco Intersight?</h3>
Cisco Intersight: The idea is to achieve the Application Continuity on the Compute level, and also on the Networking and Security level. This means that we need something to keep the coherent policies around our clouds. Intersight is a step in that direction, being the intelligent management cloud platform for Cisco Unified Computing System (UCS) and Hyper Flex<br />
<br />
<h3>
What is ACI Anywhere about?</h3>
One of the ideas of the entire Cloud native approach is to define the Policy Model, which includes Connectivity, Security and other policies, and apply the same Policy Model regardless whether your workloads are in the Public or Private cloud. In UCS we have Service Profiles, while in the ACI we have the Application Profiles. At the moment these only apply in the Data Center, where our ACI is. There are almost 4000 customers deploying ACI at the moment (January 2018). Why not extend this Policy Model to the Public Cloud? Imagine how happy the Hybrid Cloud fans, such as myself, will be when this becomes the reality.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s10.postimg.org/w9w7nrc6h/ACI_anywhere.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="457" data-original-width="800" height="182" src="https://s10.postimg.org/w9w7nrc6h/ACI_anywhere.jpg" width="320" /></a></div>
<br />
ACI Anywhere has a simple objective: take the ACI policy model to the Cloud. The new Data Center is a Multi Cloud Data Center, so what needs to be done is a consistent management paradigm across the cloud. ACI Multi Site controller allows us to control the ACI on different sites using a single management platform, using this you can choose the Tenants that are "stretched" across multiple ACIs, and the ones that are local to some of them.<br />
<br />
To sum up: <b>ACI Anywhere = ACI Multi Site + ACI in Public Cloud</b><br />
<div>
<br /></div>
How do we bring ACI Policy Model to a Public Cloud? Using the AVE (Application Virtual Edge). At the moment (ACI 3.1) AVE is basically AVS. However, it will evolve into a Virtual Gateway concept that will allow us to "put" the ACI Policy Model into any Virtual Environment, and any Public Cloud. At the moment it is announced to support VMware with NSX, OpenStack, Google Cloud, AWS and Azure. Stay tuned for more details...<br />
<br />
<h3>
What about Micro Services?</h3>
Containers are the part of the new reality, because even though not many customers are actually using them at the moment - everyone is looking at them. Kubernetes is, in accordance with that, made a crucial part of ACI. I've already tested Kubernetes and OpenShift integration with ACI in my Lab, and I must say that I'm impressed. Right after the CiscoLive in Barcelona, I will cover the details of these integrations, so follow my Tweets/Blog.<br />
<br />Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com2tag:blogger.com,1999:blog-6091645117172561542.post-21760249205357726722017-12-13T18:06:00.001+01:002018-09-06T11:40:29.485+02:00OpenStack Networking, Explanation for Humans<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Long time ago I
published an OpenStack Networking principles, you can find it here:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- <a href="https://www.snarchs.com/2015/12/openstack-neutron-and-ovs-open-virtual.html" target="_blank">OpenStack Neutron and OVS (Open Virtual Switch) translated to the Network Engineers language</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- <a href="https://www.snarchs.com/2015/12/open-virtual-switch-ovs-deep-dive-how.html" target="_blank">Open Virtual Switch (OVS) Deep Dive: How L2 Agent "wires" a new VM</a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Based
on the feedback I got, it's too complex and hard to digest, basically - not
written in a language that humans can understand. This motivated me to try and
explain it in a simpler way so that anyone, even Network engineers as myself,
could get it, ergo the name of the post.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
OpenStack is an open
source platform that is basically composed from different Projects. Networking
Project is called Neutron. To fully understand how this all comes together, I
will cover the following concepts:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Linux Networking</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">OVS (Open Virtual Switch)
Networking</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Neutron </span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Why OpenStack requires SDN.</span></li>
</ul>
<!--StartFragment-->
<!--EndFragment--><br />
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Linux Networking</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
In virtualization
network devices, such as Switches and NICs, are virtualized. virtual Network
Interface Card (vNIC) is a NIC equivalent for a Virtual Machine (VM).
Hypervisor can create one or more vNICs for each VM. Each vNIC is identical to
a physical NIC (VM doesn’t "know” that its not a physical server). </div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Switch also can be
virtualized as a virtual switch. A virtual switch works in the same way as a
physical switch, it populates the CAM table that maps different Ports to MAC
addresses. Each vNIC is connected to the vSwitch port, and<span style="mso-spacerun: yes;"> </span>these vSwitch access external physical
network through the physical NIC of Physical Server.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Before we get into
how this all comes together, we need to clarify 3 concepts:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Linux Bridge</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is a virtual Switch used
with KVM/QEMU hypervisor. Remember this, Bridge = L2 Switch, as simple as
that.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">TAP and TUN</span><span style="font-family: "calibri"; font-size: 11.0pt;"> are virtual network devices
based on Linux kernel implementation. TUN<span style="mso-spacerun: yes;">
</span>works with IP packets, TAP operates with layer 2 packets
like Ethernet frames.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">VETH
(Virtual Ethernet pair)</span><span style="font-family: "calibri"; font-size: 11.0pt;"> is created to act as virtual wiring. To connect 2 TAPs you would
need a Virtual Wire, or VETH. Essentially what you are creating is a
virtual equivalent of a patch cable. What goes in one end comes out the
other. It can be used to connect two TAPs that belong to two VMs from
different Namespaces, or to connect a Container or a VM to OVS. When VETH
connects 2 TAPs, everything that goes in on one TAP goes out on another
TAP.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
So, why the hell are
all these concepts needed, TAP, VETH, Bridge…? These are just Linux concepts
that are used to construct the Virtual Switch and give connectivity between
VMs, and between VM and the outside world. Here is how it all works:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">When you create a Linux
Bridge, you can assign TAPs to it. In order to connect the VM to this
Bridge, you need to then associate the VM vNIC to one of the TAPs.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">vNIC is associated to the TAP
programmatically, in software (When Linux bridge sends Ethernet frames to
a TAP interface, it actually is sending the bytes to a file
descriptor. Emulators like QEMU, read the bytes from this file descriptor
and pass it onto the ‘guest operating system’ inside the VM, via the
virtual network port on the VM. Tap interfaces are listed as part of
the ifconfig Linux command, if you want to make sure everything
is where it should be.)</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">OVS</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
OVS is a multilayer
virtual switch, designed to enable massive network automation through
programmatic extension. Linux Bridge can also be used as a Virtual Switch in a
Linux environment, the difference is that the Open vSwitch is targeted at
multi-server virtualization deployments where automation is used.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Open vSwitch bridge
is also used for L2 Switching, exactly like the Linux Bridge, with a pretty
important difference when it comes to Automation: it can operate in two modes:
Normal and Flow mode.</div>
<ol style="direction: ltr; font-family: Calibri; font-size: 11.0pt; font-style: normal; font-weight: bold; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="1">
<li style="font-weight: bold; margin-bottom: 0; margin-top: 0; vertical-align: middle;" value="1"><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: bold;">OVS in a “normal” mode</span><span style="font-family: "calibri"; font-size: 11.0pt; font-style: normal; font-weight: normal;">, where it acts as a normal
switch, learning and populating CAM table using ARP.</span></li>
<li style="font-weight: bold; margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">OVS
in a “flow” mode </span><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: normal;">is why we use OVS and not the Linux Bridge. It lets you
“program the flows”, using OpenFlow, OpFlex (whatever instructions come
from the SDN controller), or manually (calling ovs-ofctl add-flow).
Whatever flows are installed are used and no other behavior is implied.
Regardless how the flow is configured, it has MATCH and ACTION part.<span style="mso-spacerun: yes;"> </span>The match part of a flow defines what
fields of a frame/packet/segment must match in order to hit the flow. You
can match on most fields in the layer 2 frame, layer 3 packet or layer 4
segment. So, for example, you could match on a specific destination MAC
and IP address pair, or a specific destination TCP port. The action part
of a flow defines what is actually done on a message that matched against
the flow. You can forward the message out a specific port, drop it, change
most parts of any header, build new flows on the fly (For example to
implement a form of learning), or resubmit the message to another table.
Each flow is written to a specific table, and is given a specific
priority. Messages enter the flow table directly into table 0. From
there, each message is processed by table 0’s flows from highest to lowest
priority. If the message does not match any of the flows in table 0 it is
implicitly dropped (Unless an SDN controller is defined – In which case a
message is sent to the controller asking what to do with the received
packet).</span></li>
</ol>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<!--StartFragment-->
<!--EndFragment--></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Additional
differences between the Linux Bridge and the OVS are represented in the Table
below:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s7.postimg.cc/4rxy5z8wr/OVS_vs_Linux_Bridge.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="445" data-original-width="800" height="222" src="https://s7.postimg.cc/4rxy5z8wr/OVS_vs_Linux_Bridge.png" width="400" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Open vSwitch can
operate both as a soft switch running within the hypervisor, and as the control
stack for switching silicon (physical switch).</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">OpenStack Networking and Neutron</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
One of the mail
features that OpenStack brings to the table is Multi Tenancy. Therefore, the
entire platform needs to support a Multi Tenant architecture, including
Networking. This means that different Tenants should be allowed to use the
overlapping IP Spaces and Overlapping IP Addresses should be allowed in
different Tenants. This is enabled using the following technologies:</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Network
Namespaces</span><span style="font-family: "calibri"; font-size: 11.0pt;">,
which are, in a networking language, equal to the VRFs.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Tenant
Networks</span><span style="font-family: "calibri"; font-size: 11.0pt;"> are
owned and managed by the tenants. These networks are internal to the
Tenant, and every Tenant is basically allowed to use any IP addressing
space they want.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">Provider
Networks</span><span style="font-family: "calibri"; font-size: 11.0pt;"> are
networks created by Administrators to map to physical network in data
center. They are used to publish services from particular Tenants, or to
allow OpenStack VMs (called Instances) to go out of the OpenStack Tenant
environment.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
To understand the
concept of Provider Networks, I'll explain the two types of Provider Networks,
as the only possible way of VMs to achieve the connectivity to the outside
network.</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">SNAT (Source NAT) is similar
to a NAT service an Office uses on a Firewall to go out to the Internet.
All the VMs can use a single IP (or a group of IPs) that Admin configured
when deploying OpenStack, to get to the Network outside of OpenStack
environment (Internet, or a LAN network).</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt;">Floating IP is used for
publishing services. To all the VMs you need to be accessible from the
outside, you will manually need to assign a Floating IP.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Bridges and Bridge
Mappings are a crucial concept when it comes to OpenStack Networking, it’s all
about how different BRIDGES and TAPs come together.</div>
<ul style="direction: ltr; margin-bottom: 0in; margin-left: .375in; margin-top: 0in; unicode-bidi: embed;" type="disc">
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">br-int</span><span style="font-family: "calibri"; font-size: 11.0pt;">, integration network bridge,
is used to connect the virtual machines to internal tenant networks.</span></li>
<li style="margin-bottom: 0; margin-top: 0; vertical-align: middle;"><span style="font-family: "calibri"; font-size: 11.0pt; font-weight: bold;">br-ext</span><span style="font-family: "calibri"; font-size: 11.0pt;">, external network bridge, is
used for the connection to the PROVIDER networks, to enable connectivity
to and from virtual instances. Br-ext is mapped to a Physical Network, and
this is where the Floating IP and SNAT IP addresses will be assigned to
the instances going out from the OpenStack via the Provider Networks.</span></li>
</ul>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<!--StartFragment-->
<!--EndFragment--></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Let's check the Data
Flow now, on an example of a single OpenStack instance (Instance-1) being
assigned a Floating IP and accessing the Public Network.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s7.postimg.cc/6wib75pzv/OVS_example.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="418" data-original-width="800" height="208" src="https://s7.postimg.cc/6wib75pzv/OVS_example.png" width="400" /></a></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
As previously
explained, a NAT is done on a br-ex, so the Floating IP is also assigned on a
br-ex, and from that point on the Instance is accessing the Public Networks
with the assigned Floating IP. In case the Floating IP has not been assigned,
the Instance is accessing the outside world using the SNAT.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<span style="font-weight: bold;">Why OpenStack requires SDN</span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
As explained before,
Neutron is an OpenStack with an API for defining network configuration. It
offers multi-tenancy with self-service. Neutron uses plugins for L2
connectivity, IP address management, L3 routing, NAT, VPN and Firewall
capabilities.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
Here is why SDN is an essential requirement for any OpenStack production deployment:</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- OpenStack cannot configure a physical network in accordance with it's needs to interconnect VMs in different Compute nodes.</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- Neutron does the basic networking correctly. It cannot do routing the correct way, security policies, HA of the external connectivity, network performance management etc.</div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 11pt;">- </span><span style="font-size: 14.6667px;">OpenStack Neutron defines services for a VM provisioning within an OpenStack deployment, these services include: NAT, DHCP, Metadata, etc. All of these services have to be highly available and scalable to meet environment’s demands. </span></span></div>
<div style="margin: 0in;">
<span style="font-family: "calibri";"><span style="font-size: 11pt;">- </span><span style="font-size: 14.6667px;">SDN reduces Load on Neutron.</span></span></div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
- Last, but not the least, I've never seen a production OpenStack deployment with no SDN. Just saying...</div>
<div style="font-family: Calibri; font-size: 11.0pt; margin: 0in;">
<br /></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com1tag:blogger.com,1999:blog-6091645117172561542.post-1628032531464552982017-06-30T19:57:00.000+02:002018-06-15T16:12:15.919+02:00On PaloAlto and NSX IntegrationThe VM-Series firewall for VMware NSX is jointly developed by Palo Alto Networks and VMware. NetX APIs are used to integrate the Palo Alto Networks next-generation firewalls and Panorama with VMware ESXi servers. Before getting into the technical part, make sure you understand what NSX is and how micro segmentation is deployed, what the difference between the Distributed Firewall and a traditional Firewall that protects the perimeter is. You can check out some of my previous posts in the <a href="https://www.snarchs.com/p/blogmap.html" target="_blank">Blog Map</a>.<br />
<br />
The idea is to deploy the Palo Alto Networks firewall as a service on a cluster of VMware ESXi servers where the NSX has been enabled. The objective is to protect the East-West traffic in your VMware environment and "steer" the FW rules between the NSX "native" Firewall and the Palo Alto Firewall. We are doing this integration in order to be able to later enforce different type of Security Policies depending on whether we want to protect the traffic within the VMs of the same Tier (intra-tier), or between different Tiers (Inter-Tier). Best practice would be:<br />
<br />
<ul>
<li><b>Inter-tier traffic (Web server to App or DB server) is protected by PaloAlto</b> Networks VM-series firewall which provides advanced security capabilities with its single pass architecture in the form of App-ID, Content-ID, and User-ID. On a diagram below a PA NGFW is protecting the traffic between the HRs Web and DB Servers.</li>
<li><b>Intra-tier traffic (web server to web server) is protected by NSX DFW</b> which provides near line rate performance for L2-L4 security functions. On a diagram below, NSX DFW is protecting the traffic between the two HRs Web servers.</li>
</ul>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s23.postimg.cc/zd74ogld7/Steering_Rules.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="639" data-original-width="800" height="318" src="https://s23.postimg.cc/zd74ogld7/Steering_Rules.png" width="400" /></a></div>
<h3>
<b>Components</b></h3>
<br />
Before we proceed with the detailed explanation of how to deploy and configure the environment, let's clear out what are the components of the VM-Series for NSX Solution, how they work together and what are the benefits. The components of the integrated solution are the following:<br />
<br />
<ul>
<li><b>vCenter Server</b>, the centralized management tool for the vSphere suite. The vCenter server is required to manage the NSX Manager and the ESXi hosts in your data center. This joint solution requires that the ESXi hosts be organized into one or more clusters on the vCenter server and must be connected to a distributed virtual switch.</li>
<li><b>NSX Manager</b>, the VMware Networking and Security platform, or simply said - SDN. The NSX Firewall and the Service Composer are key features of the NSX Manager. The NSX firewall is a logical firewall that allows you to attach network and security services to the virtual machines, and the Service Composer allows you to group virtual machines and create policy to redirect traffic to the VM-Series firewall</li>
<li><b>Panorama</b>, centralized management tool for the Palo Alto NGFW (Next Generation Firewalls). In this solution, <b>Panorama works with the NSX Manager to deploy, license, and centrally administer configuration and policies on the VM-Series firewalls for NSX</b>. Panorama is used to register the VM-Series firewall for NSX as the Palo Alto Networks NGFW service on the NSX Manager. This allows the NSX Manager to deploy the VM-Series firewall for NSX on each ESXi host in the ESXi cluster. When a new VM-Series firewall is deployed in NSX, it communicates with Panorama to obtain the license and receives its configuration/policies from Panorama. Panorama must be able to connect to the NSX Manager, the vCenter server, the VM-Series firewalls and the Palo Alto Networks update server.</li>
<li><b>VM-Series Firewall for NSX</b> (VM-100, VM-200, VM-300, VM-500, and VM-1000-HV, support NSX). The VM-Series firewall for NSX is the VM-Series firewall that is deployed on the ESXi hypervisor. The integration with the NetX API makes it possible to automate the process of installing the VM-Series firewall directly on the ESXi hypervisor, and allows the hypervisor to forward traffic to the VM-Series firewall without using the vSwitch configuration. The VM-Series firewall for NSX only supports virtual wire interfaces. On this firewall, ethernet 1/1 and ethernet 1/2 are bound together through a virtual wire and use the NetX dataplane API to communicate with the hypervisor. Layer 2 or Layer 3 interfaces are neither required nor supported on the VM-Series firewall for NSX, and therefore no switching or routing actions can be performed by the firewall.</li>
</ul>
<br />
<br />
Ports/Protocols you need to enable for the Network Communication:<br />
<br />
<ul>
<li>Panorama: To obtain software updates and dynamic updates, Panorama uses SSL to access updates.paloaltonetworks.com on TCP/443; this URL leverages the CDN infrastructure. If you need a single IP address, use staticupdates.paloaltonetworks.com.</li>
<li>The NSX Manager and Panorama use SSL to communicate on TCP/443.</li>
<li>VM-Series Firewall for NSX: If you plan to use Wildfire, the VM-Series firewalls must be able to access wildfire.paloaltonetworks.com on port 443. This is an SSL connection and the App-ID is PaloAlto-wildfire-cloud.</li>
<li>The management interface on the VM-Series firewall uses SSL to communicate with Panorama over TCP/3789.</li>
<li>vCenter Server The vCenter Server must be able to reach the deployment web server that is hosting the VM-Series OVA. The port is TCP/80 by default or App-ID web-browsing.</li>
</ul>
<br />
<br />
Which version of Panorama, vSphere, NSX and PA-VM should I use?<br />
<br />
<ul>
<li>Panorama: For a long time the VM-1000-HV was the only available PaloAlto VM Firewall for this integration. Don't get me wrong, it's a great option, but if cost of the solution is something that you might worry you - I've got some good news. Since Panorama 8.0 all the PA-VM versions are supported (VM-100, VM-300, VM-500 and of course VM-1000). It gets even better - you can start with the VM-100, and upgrade from there if you need more capacity in the future.</li>
<li>NSX: For my lab I used NSX 6.2.5. I recommend you go directly for 6.3.x, all the concepts explained here apply.</li>
</ul>
<div>
<br /></div>
<h3>
<b>Integration</b></h3>
Now that we know the components, let's see how it all fits together. NSX Manager, ESXi servers and Panorama work together to automate the deployment of the VM-Series firewall, as shown in the diagram below. Lets get deeper into this...<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s17.postimg.cc/fme61tzcv/Virtualization-112.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="353" data-original-width="461" height="306" src="https://s17.postimg.cc/fme61tzcv/Virtualization-112.jpg" width="400" /></a></div>
<br />
<br />
<br />
<br />
<h4>
1.1 Install the VMware NSX Plugin</h4>
Before you start the integration, you need to make sure that your NSX is operational, NSX Controllers in the "Connected" state (vSphere > NSX > Installation > Management). I strongly advise you to upgrade your Panorama to 8.0.x, if you haven't already. In my Lab I used only 2 hosts at first. Once I had everything fully functional, I added the other hosts.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s3.postimg.cc/f5vqbpp43/NSX_Installed_in_Hosts.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="155" data-original-width="800" height="62" src="https://s3.postimg.cc/f5vqbpp43/NSX_Installed_in_Hosts.png" width="320" /></a></div>
<br />
You need to Download the Plugin from here (you will need a Palo Alto Support account):<br />
https://support.paloaltonetworks.com/Updates/SoftwareUpdates/1904<br />
<br />
Log in to Panorama, and go to "Panorama Tab > Plugins". Upload the Plugin, and press "Install". A new "VMware NSX" sub-menu will appear on the left, as shown below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s24.postimg.cc/j3h3k2hat/Panorama_NSX_Plugin.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="341" data-original-width="800" height="170" src="https://s24.postimg.cc/j3h3k2hat/Panorama_NSX_Plugin.png" width="400" /></a></div>
<br />
<br />
Next you need to set up access to the NSX Manager. Select Panorama > VMware NSX > Service Managers and click Add. Enter the Service Manager Name and the other required info. If you do this step correctly, on the NSX Manager, this name will be displayed in the Service Manager column on Networking & Security > Service Definitions > Service Managers.<br />
<br />
IMPORTANT: The ampersand (&) special character is not supported in the NSX manager account password. If a password includes an ampersand, the connection between Panorama and NSX manager fails.<br />
<br />
TIP: Once the Services are Synchronized, in PANOS 8.0 you won´t be able to see the Service Manager Status. Don´t panic, this is Ok. As long as you see the new Service Manager has been configured in NSX (Networking & Security > Service Definitions > Service Managers) - you´re good to go.<br />
<br />
In Panorama you will also see a new Administrator user called "__vmware_nsx" has been configured. In NSX try to edit the newly created "Service Manager". You will notice that the credentials are associated to this new user.<br />
<br />
<h4>
1.2 Create Template(s) and Device Group(s) on Panorama</h4>
To manage the VM-Series firewalls for NSX using Panorama, the firewalls must belong to a device group and a template. Device groups allow you to assemble firewalls that need similar policies and objects as a logical unit; the configuration is defined using the Objects and Policies tabs on Panorama. Use templates to configure the settings that are required for the VM-Series firewalls to operate on the network and associate; the configuration is defined using the Device and Network tabs on Panorama (Groped as Templates). And each template containing zones used in your NSX configuration on Panorama must be associated with a service definition; at a minimum, you must create a zone within the template so that the NSX Manager can redirect traffic to the VM-Series firewall.<br />
<br />
Go to Panorama > Device Groups, and click Add. Name your Device Group something Intuitive, like NSX Firewalls. After the firewalls are deployed and provisioned, they will display under Panorama > Managed Devices and will be listed in the device group.<br />
<br />
Now add a template or a template stack. Select Panorama > Templates, and click Add. After this you need to create the Zone for each template (be sure to set the interface Type to Virtual Wire.). Panorama creates a corresponding service profile on NSX Manager for each qualified zone upon commit.<br />
<br />
IMPORTANT: For a single-tenant deployment, create one zone. If you have multi-tenant deployment, create a zone for each sub-tenant.<br />
<br />
Now you need to add a new Service Definition. This is basically used for Panorama to know how to provision a PaloAlto Firewall on the Hosts where it is needed. Select Panorama > VMware NSX > Service Definitions.<br />
<br />
<b>TIP</b>: Before you define the Service Definition, you need to place your PA-XXX.ova file on a Web Server. I know, not as cool as the Architects of the solution imagined it, but still... it´s logical that Panorama needs an Image Repository with different types of PA-VM, because a big environment might require a variety of different Firewalls.<br />
<br />
Once the Service Definition is created, Select Panorama > VMware NSX > Service Manager and click the link of the service manager name. Under Service Definitions, click Add and select your service definition from the drop-down.<br />
<br />
Now you need to Add the authorization code to license the firewalls. I hope you already have the Auth Code by now. Select Panorama > Device Groups and choose the device group you associated with the service definition you just created. Under Dynamically Added Device Properties, add the authorization code you received with your order fulfillments email and select a PAN-OS software version from the SW Version drop-down. When a new firewall is deployed under NSX and added to the selected device group, the authorization code is applied and the firewall is upgraded to the select version of PAN-OS.<br />
<br />
<b>IMPORTANT: </b>You need to Install a License Deactivation API Key in Panorama before you proceed with the FW Deployment in the ESXi Cluster. This is important before you want your Panorama to take care of the Licenses using the Auth-Code.<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">admin@Panorama> request license api-key set key bea265bdb4c832793b857cfa1bf047845dc82e3b3c1b18c1b2e59796147340eb</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">API Key is successfully set</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">admin@Panorama></span><br />
<br />
<h4>
2. Register the VM-Series Firewall as a Service on the NSX Manager</h4>
2.1 The first step is to register the Palo Alto Networks NGFW as a service on the NSX Manager. The registration process uses the NetX management plane API to enable bi-directional communication between Panorama and the NSX Manager. Panorama is configured with the IP address and access credentials to initiate a connection and register the Palo Alto Networks NGFW service on the NSX Manager. The service definition includes the URL for accessing the VM-Series base image that is required to deploy the VM-Series firewall for NSX, the authorization code for retrieving the license and the device group and template to which the VM-Series firewalls will belong. The NSX manager uses this management plane connection to share updates on the changes in the virtual environment with Panorama.<br />
<br />
2.2 Deploy the VM-Series automatically from NSX —The NSX Manager collects the VM-Series base image from the URL specified during registration and installs an instance of the VM-Series firewall on each ESXi host in the ESXi cluster. From a static management IP pool or a DHCP service (that you define on the NSX Manager), a management IP address is assigned to the VM-Series firewall and the Panorama IP address is provided to the firewall. When the firewall boots up, the NetX data plane integration API connects the VM-Series firewall to the hypervisor so that it can receive traffic from the vSwitch.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s21.postimg.cc/6pb4o4r7r/Virtualization-113_traffic_flow.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="371" data-original-width="399" height="297" src="https://s21.postimg.cc/6pb4o4r7r/Virtualization-113_traffic_flow.jpg" width="320" /></a></div>
<br />
<br />
2.3 Establish communication between the VM-Series firewall and Panorama : The VM-Series firewall then initiates a connection to Panorama to obtain its license. Panorama retrieves the license from the update server and pushes it to the firewall. The VM-Series firewall receives the license (VM-1000-HV) and reboots with a valid serial number.<br />
<br />
2.4 Install configuration/policy from Panorama to the VM-Series firewall : The VM-Series firewall reconnects with Panorama and provides its serial number. Panorama now adds the firewall to the device group and template that was defined in the service definition and pushes the configuration and policy rules to the firewall. The VM-Series firewall is now available as a security virtual machine that can be further configured to safely enable applications on the network.<br />
<br />
2.5 Push traffic redirection rules to NSX Manager : On Panorama, create security groups and define network introspection rules that specify the guests from which traffic will be steered to the VM-Series firewall. See Integrated Policy Rules for details.<br />
<br />
2.6 Receive real-time updates from NSX Manager : The NSX Manager sends real-time updates on the changes in the virtual environment to Panorama. These updates include information on the security groups and IP addresses of guests that are part of the security group from which traffic is redirected to the VM-Series firewall. See Integrated Policy Rules for details.<br />
<br />
2.7 Use dynamic address groups in policy and push dynamic updates from Panorama to the VM-Series firewalls : On Panorama, use the real-time updates on security groups to create dynamic address groups, bind them to security policies and then push these policies to the VM-Series firewalls. Every VM-Series firewall in the device group will have the same set of policies and is now completely marshaler to secure the SDDC. See Policy Enforcement using Dynamic Address Groups for details.<br />
<br />
<h4>
3. Create Steering Rules</h4>
IMPORTANT: The default policy on the VM-Series firewall is set to deny all traffic, which means that all traffic redirected to the VM-Series firewall will be dropped. Have this in mind before you activate PA NGFW in your VMware environment.<br />
<br />
Panorama serves as the single point of configuration that provides the NSX Manager with the contextual information required to redirect traffic from the guest virtual machines to the VM-Series firewall. The traffic steering rules are defined on Panorama and pushed to NSX Manager; these determine what traffic from which guests in the cluster are steered to the Palo Alto Networks NGFW service. Security enforcement rules are also defined on Panorama and pushed to the VM-Series firewalls for the traffic that is steered to the Palo Alto Networks NGFW service.<br />
<br />
Steering Rules —The rules for directing traffic from the guests on each ESXi host are defined on Panorama and applied by NSX Manager as partner security services rules.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s18.postimg.cc/cyhgh3mqx/Virtualization-116_redirection.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="343" data-original-width="650" height="168" src="https://s18.postimg.cc/cyhgh3mqx/Virtualization-116_redirection.jpg" width="320" /></a></div>
<br />
For traffic that needs to be inspected and secured by the VM-Series firewall, the steering rules created on Panorama allow you to redirect the traffic to the Palo Alto Networks NGFW service. This traffic is then steered to the VM-Series firewall and is first processed by the VM-Series firewall before it goes to the virtual switch.<br />
<br />
Traffic that does not need to be inspected by the VM-Series firewall, for example network data backup or traffic to an internal domain controller, does not need to be redirected to the VM-Series firewall and can be sent to the virtual switch for onward processing.<br />
<br />
Rules centrally managed on Panorama and applied by the VM-Series firewall —The next- generation firewall rules are applied by the VM-Series firewall. These rules are centrally defined and managed on Panorama using templates and device groups and pushed to the VM-Series firewalls. The VM-Series firewall then enforces security policy by matching on source or destination IP address—the use of dynamic address groups allows the firewall to populate the members of the groups in real time—and forwards the traffic to the filters on the NSX Firewall.<br />
<br />
<h3>
<b>Policy Enforcement using Dynamic Address Groups</b></h3>
Unlike the other versions of the VM-Series firewall, because both virtual wire interfaces (and subinterfaces) belong to the same zone, the VM-Series firewall for NSX uses dynamic address groups as the traffic segmentation mechanism. A security policy rule on the VM-Series firewall for NSX must have the same source and destination zone, therefore to implement different treatment of traffic, you use dynamic address groups as source or destination objects in security policy rules.<br />
<br />
Dynamic address groups offer a way to automate the process of referencing source and/or destination addresses within security policies because IP addresses are constantly changing in a data center environment. Unlike static address objects that must be manually updated in configuration and committed whenever there is an address change (addition, deletion, or move), dynamic address groups automatically adapt to changes.<br />
<br />
Any dynamic address groups created in a device group belonging to NSX configuration and configured with the match criterion _nsx_<dynamic address="" group="" name=""> trigger the creation on corresponding security groups on the NSX Manager. In an ESXi cluster with multiple customers or tenants, the ability to filter security groups for a service profile (zone on Panorama) on the NSX Manager allows you to enforce policy when you have overlapping IP addresses across different security groups in your virtual environment.</dynamic><br />
<br />
If, for example, you have a multi-tier architecture for web applications, on Panorama you create three dynamic address groups for the WebFrontEnd servers, Application servers and the Database servers. When you commit these changes on Panorama, it triggers the creation of three corresponding security groups on NSX Manager.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s2.postimg.cc/tqwxpvop5/Virtualization-117_dynamic_address_groups.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="517" data-original-width="642" height="321" src="https://s2.postimg.cc/tqwxpvop5/Virtualization-117_dynamic_address_groups.jpg" width="400" /></a></div>
<br />
<br />
CONCLUSION: Panorama Dynamic Address Group = NSX Security Group<br />
<br />
On NSX Manager, you can then add guest VMs to the appropriate security groups. Then, in security policy you can use the dynamic address groups as source or destination objects, define the applications that are permitted to traverse these servers, and push the rules to the VM-Series firewalls.<br />
<br />
Each time a guest is added or modified in the ESXi cluster or a security group is updated or created, the NSX Manager uses the PAN-OS REST-based XML API to update Panorama with the IP address, and the security group to which the guest belongs.<br />
<br />
When Panorama receives the API notification, it verifies/updates the IP address of each guest and the security group and the service profile to which that guest belongs. Then, Panorama pushes these real-time updates to all the firewalls that are included in the device group and notifies device groups in the service manager configuration on Panorama.<br />
<br />
On each firewall, all policy rules that reference these dynamic address groups are updated at runtime. Because the firewall matches on the security group tag to determine the members of a dynamic address group, you do not need to modify or update the policy when you make changes in the virtual environment. The firewall matches the tags to find the current members of each dynamic address group and applies the security policy to the source/destination IP address that are included in the group.<br />
<br />
Is this a Multi Tenant environment? For enabling traffic separation in a multi-tenancy environment, you can create additional zones that internally map to a pair of virtual wire sub-interfaces on the parent virtual wire interfaces, Ethernet 1/1 and Ethernet 1/2.<br />
<div>
<br /></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com1tag:blogger.com,1999:blog-6091645117172561542.post-66774035747146372242017-06-24T10:44:00.002+02:002018-06-15T16:14:06.838+02:00Nuage Networks VSP Deep DiveEver since Cisco bought Insieme and created Cisco ACI, and VMware bought Nicira and created NSX, I've been intensively deep-diving and blogging about both these solutions, how they compare to each other and to some Open Source SDN solutions out there, such as OpenDayLight and Open Contrail |(check out the Blog Map section for some of my older posts). I even did boot camps and got the highest certifications in both NSX and ACI. SDN is still a rather new technology, and I wanted to make sure I have enough expertise to always explain to a customer which SDN solution is the right one for their Organization and why. Apart from ACI, NSX and open source solutions, there is another player on the SDN market, and from what I've seen - they mean business! I'm talking about Nuage Networks, acquired by Nokia from Alcatel-Lucent in November 2016. Even though I've known about this solution for a while, my opinion was that their strongest side was marketing, so I didn’t spend a lot of time investigating about Nuage (it's also pretty difficult to find the information about Nuage, there is an apparent lack of experts/blogs/technical info about the product). I finally decided to give them an opportunity, I did a boot camp, a lots of Hands-on, and recently I passed a certification 4A0-N01 Nuage Network Professional – Datacenter (NNP-DC). Let me share with you what Nuage Networks is all about, and give the unbiased opinion about their understanding of SDN, and how they compare to other SDN solutions on the market.<br />
<br />
Disclaimer: Some of the materials I used come directly from Nuage technical documentation, which is for some reason not available to the public (and it should be!). If someone from Nokia is reading this, please note that revealing technical information about what your product provides more market, and gives more visibility to your product. I strongly advise you to make as much Nuage documentation public as possible, because if your product is good (and in my opinion - it is), invite bloggers and technical experts to give you feedback, if they feel comfortable with your product, they will feel free sharing it with potential customers.<br />
<br />
Before I get deeper into what Nuage VSP is good for, let's make sure we understand the difference between the IaaS, PaaS and SaaS. In order to really get what your company is doing (or should be doing), whether you are a Service Provider (SP), or an Enterprise consuming resources provided by other Service Provider, you need to have a clear distinction of what is handled by whom in each of these architectures. Basically:<br />
<br />
<ul>
<li><b>IaaS</b> (Infrastructure as a Service) - SP Provides Network, Compute and Storage, Customer builds OS and Apps</li>
<li><b>PaaS</b> (Platform as a Service) - SP also Provides the OS. Who takes tare of the OS Upgrades and other stuff? Good questions… Depends on the PaaS Provider, could be either way.</li>
<li><b>SaaS</b> (Software as a Service) - SP owns everything, including the application</li>
</ul>
<br />
Let's start with the basics. We already know what SDN is all about, separating the Control Plane from the Data Plane, and providing a single Management plane that exposes the Northbound APIs. Nuage follows the same concepts. Nuage created a platform called VSP. VSP stands for Virtualized Services Platform, and it does the Orchestration of the Deployment, handling the following Planes:<br />
<br />
<ul>
<li>Management plane, represented by Nuage Virtual Service Directory (VSD) and the Cloud Management System or CMS (OpenStack, CloudStack etc.)</li>
<li>Control plane, handled by Nuage Service Controller (VSC)</li>
<li>Data Plane, handled by a Virtual Router & Switch (VRS)</li>
</ul>
<br />
<div>
<br /></div>
<div>
<a href="https://s3.postimg.cc/bdo0a8ew3/Screenshot_2017-06-20_20.19.57.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="744" data-original-width="522" height="320" src="https://s3.postimg.cc/bdo0a8ew3/Screenshot_2017-06-20_20.19.57.png" width="224" /></a></div>
<div>
<br /></div>
<div>
<div>
VSP includes the software suite comprising of three key products:</div>
<div>
<br />
<ul>
<li>VSD (Virtual Services Directory), which holds the policy and network service templates.</li>
<li>VSC (Virtual Services Controller), which is the SDN controller that communicates to the hypervisors.</li>
<li>VRS (Virtual Routing and Switching) agent that resides within the hypervisor on the server hardware.</li>
</ul>
</div>
<div>
<br /></div>
<div>
Let's now take a deep dive into what communication protocols are deployed between different VSP components:</div>
<div>
<br />
<ul>
<li>Communication between the CMS (Cloud Management System, such as OpenStack, CloudStack, vCenter, vCloud, etc.) and the VSD is done via RESTful APIs. We're talking about the Northbound APIs that allow us to configure Nuage Platform, or VSP.</li>
<li>Communication between the VSD and VSC is via industry standard XMPP (Extensible Messaging and Presence Protocol), using the Management network. SSL is optional, but recommendable.</li>
<li>Communication between the VSC and the hypervisors (including the VRS) is via OpenFlow, using the Underlay Network. SSL is again, optional but recommended.</li>
<li>SDN is all about virtualization, but luckily - physical servers have not been forgotten. To integrate “bare metal” assets such as non-virtualized servers and appliances, Nuage Networks also provides a comprehensive Gateway solution: software-based VRS gateway (VRS-G) and hardware-based 7850 VSG.</li>
</ul>
</div>
<div>
<br /></div>
<div>
I'd recommend you to get acquainted with the individual components of the architecture by reading the rest of this post first, and then re-visit the previous paragraph. It will all make much more sense for you.</div>
<div>
<br /></div>
<div>
Let's now check out individual Nuage VSP components, and see what each one does. Once again, I'll try to be methodical (not an intuitive task for my mind), and try to structure the post, so that you can follow:</div>
<div>
<br />
<ol>
<li>VSD, or the Virtualized Services Directory at the Network Management Plane</li>
<li>VSC, or the Virtualized Services Controller, at the Network Control Plane</li>
<li>VRS and VSG , or the Virtualized Routing & Switching and Virtualized Services Gateway, at the network Data Plane</li>
<li>Security Policies: NFV and Service Chaining</li>
</ol>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b>1. VSD - Virtualized Services Directory</b>, holds the Policy and Network Templates. VSD uses XMPP protocol to communicate with VSC.</div>
<div>
<br /></div>
<div>
VSD is where we do the Service Definition by defining Network Service Templates. The service definition includes domain, zone, subnet and policy templates. A domain template can also include policies (e.g. security, forwarding, QoS, etc.) to be applied at the different levels (vPort, subnet, zone, domain). I will cover all these concepts in just a while. It´s an essential component that will manage everything. It can be deployed as a Physical or a Virtual machine. it comes as an OVA file (for ESXi) and QCOW2 file (for KVM), or as an ISO image (recommended for the production environment). You can choose whether you want to do a standalone deployment, or a cluster of 3 VMs. To work properly, VSD requires an NTP server and a DNS server in the network.</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s3.postimg.cc/xa6ookd0z/Screenshot_2017-06-17_18.03.13.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="388" data-original-width="800" height="193" src="https://s3.postimg.cc/xa6ookd0z/Screenshot_2017-06-17_18.03.13.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<div>
The VSD also contains a powerful analytics engine (optional, based on Elastic Search). The VSD supports RESTful APIs for communicating to the cloud provider’s management systems. In the case of OpenStack, it is between nova and nova-compute, while vCloud uses the vCenter API to access the ESXi HVs.</div>
<div>
<br /></div>
<div>
VSD has two types of users:</div>
<div>
Administrator/CSP Users, who will have full visibility into all of the functionality of VSD</div>
<div>
Enterprise/Organization Users. An enterprise user belongs to one, and only one, specific enterprise.</div>
<div>
<br /></div>
<div>
<b>TIP</b>: Have in mind that if you're interesting in LDAP, users must be manually created in VSD, even if they have already been created in the LDAP directory.</div>
<div>
<br /></div>
<div>
VSD Service Abstraction - a VSD way of creating an Object Tree, where the Domain is the one Root and Zones, Subnets and another Object have an exact place in the Tree. VSD then translates the Service Abstraction into the Service Instances, following the same Object Tree. Having in mind that domain is mapped to a distributed VPRN instance (dVPRN) while a subnet is mapped to a distributed RVPLS instance (dRVPLS), we are counting with:</div>
<div>
<br />
<ul>
<li>L2 Service Instances (vRVPLS)</li>
<li>L3 Service Instances (dRVPN)</li>
</ul>
</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s3.postimg.cc/pl061yqqb/Screenshot_2017-06-18_11.34.21.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="778" data-original-width="726" height="320" src="https://s3.postimg.cc/pl061yqqb/Screenshot_2017-06-18_11.34.21.png" width="298" /></a></div>
<div>
<br /></div>
<div>
<div>
<b>Domain</b>: An enterprise contains one or more domains. A domain is a single “Layer 3” space, which can include one or more subnetworks that can communicate with each other. In standard networking terminology, a domain maps to a VPRN (Virtual Private Routed Network) service instance. Route distinguisher (RD) and route target (RT) values for the VPRN service are generated automatically by default, but can be modified. CSP Root users can create domain template for all the enterprises. Enterprise Administrators and Network Designers group users can create domain templates for their enterprises. Users that belong to other groups cannot create domain templates.</div>
<div>
<b>Layer 2 Domain</b>: A standard domain is a Layer 3 construct, including routing between subnets. A Layer 2 domain, however, is a mechanism to provide a single subnet, or a single L2 broadcast domain within the datacenter environment. It is possible to extend that broadcast domain into the WAN, or legacy VLAN.</div>
<div>
<b>Zone</b>: Zones are defined within a domain. A zone does not map to anything on the network directly, but instead it acts as an object with which policies are associated such that all endpoints in the zone adhere to the same set of policies.</div>
<div>
<b>Subnet</b>: Subnets are defined within a zone. A subnet is a specific IP subnet within the domain instance. The subnet is instantiated as a routed virtual private LAN service (R‐VPLS). A subnet is unique and distinct within a domain; that is, subnets within a domain are not allowed to overlap or to contain other subnets in accordance with the standard IP subnet definitions.</div>
<div>
<b>vPorts</b>: Intended to provide more granular configuration than at the subnet level, and also support a split workflow. The vPort is configured and associated with a VM port (or gateway port) before the port exists on the hypervisor or gateway. Ports that connect the Bare Metal Servers to an Overlay are also called vPort. Whenever an vPort is instantiated, an IP address is assigned to it, unique at the Domain level, from the Subnet that the vPort belongs to. VSD is responsible for assigning the correct IP address, regardless if the VM asks for a specific IP (statically configured on the OS), or from a DHCP pool. The same Virtual IP can be assigned to multiple vPorts for redundancy (must be different then any of the IPs assigned to the vPorts).</div>
<div>
All ports will have a corresponding vPort, either auto-configured or configured via REST API. Configuration attributes may optionally be configured on the vPort.</div>
<div>
<br /></div>
<div>
VM is formed from its profile, which contains the VM metadata. This metadata defines which Domain, Zone, Subnet and vPort to apply to every vNIC of the VM. It also defines which Enterprise and User Group it belongs to. Additionally, some metadata may be specified if attaching to a specific vPort is required. When a new VM is created, a VM creation request is sent to the VSC from the VRS agent in an OpenFlow message using the Underlay Network. This message contains the VM-related metadata. VSC forwards the request one level higher in the hierarchy, to the VSD in an XMPP message using the Management Network. The VSD receives the VM creation request, reads its metadata and checks them against the policy definitions. The VSD learns the MAC address assigned to this VM from the metadata, and in a VSD managed IP address allocation scenario, it assigns an IP address for it from the subnet (usually the next available IP address).</div>
<div>
<br /></div>
<div>
VSD has a somewhat complex architecture. The components of the VSD can be centralized on a single machine or distributed across multiple machines for redundancy and scale. Some of the most important to have in mind at this point are:</div>
<div>
<br />
<ul>
<li>TNC stands for trusted network connect, which is an open architecture for network access control.</li>
<li>Policy management engine evaluates the policy rules configured on the VSD (Security and QoS policies, IP assignments etc.) It sends policies to VSC based on network events.</li>
<li>VSD mediator is a VSD Southbound interface used for communication to the VSC. It receives requests for policy information and updates from the VSC, and pushes policy updates to the VSC. The VSD itself is an XMPP client: it communicates with an XMPP server, or server clusters.</li>
<li>Statistics engine collects fine-grained network information at the VRS, VSC and VM levels. It can collect various packet-based statistics such as Packets in/out, dropped packets in/out, dropped by rate limit etc. It provides an open interface for Nuage and third-party analytics applications. Have in mind that by default, Statistics collection is disabled on the VSD. A separate VSD node running Elastic Search needs to be deployed (can also be deployed as a Cluster).</li>
<li>REST API is the VSD Northbound interface, which exposes all the VSD functionalities via API calls. It can be used by Nuage CMS plug-ins for integration with many CMSs.</li>
</ul>
</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s22.postimg.cc/9s6krigk1/Screenshot_2017-06-20_20.39.17.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="493" data-original-width="800" height="246" src="https://s22.postimg.cc/9s6krigk1/Screenshot_2017-06-20_20.39.17.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<b>2. VSC - Virtualized Services Controller</b> - SDN Controller, controls the Network, communicates with the Hypervisor and collects the VM related information such as MAC and IP addresses . VSC uses OpenFlow to control the VRS. On each VRS we need to define which VSC is active, and which is standby (you can configure various active VCS for Load Balancing). OpenFlow uses TCP port 6633, and it is used to download actual L2/L3 FIBs to the virtual switch components on the Hypervisor.</div>
<div>
<br /></div>
<div>
<a href="https://s21.postimg.cc/yyly2hcd3/Screenshot_2017-06-17_18.07.08.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="426" data-original-width="800" height="212" src="https://s21.postimg.cc/yyly2hcd3/Screenshot_2017-06-17_18.07.08.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<div>
VSC is only installed as a VM (or as an Integrated Module on a Nuage NSG, when NSG is used as VxLAN Gateway), and it comes as OVA file, a QCOW2 file and a VMDK file. VSC has a control interface connected to the Underlay. It is based on Nokia Service Router Operating System (SROS), which is somewhat similar to Cisco IOS (not the same commands, but… intuitive, if you come from Cisco).</div>
<div>
<br /></div>
<div>
Now comes a really cool part about why Nuage. Controllers act like Router Control Plane, and routing is established between VSCs and other routers. This makes is so much easier to implement DCI. VSC needs a routing protocol to exchange the routes with the other VSCs. It can be ISIS, OSPF or Static Routes. MP-BGP EVPN also needs to be established between all the VSCs.</div>
<div>
<br /></div>
<div>
The VSC has three main communication directions:</div>
<div>
Northbound: to the VSD via XMPP</div>
<div>
East/West: federation functions to other VSCs or IP/MPLS Provider Edge nodes via MP-BGP</div>
<div>
Southbound: to the VRSs via OpenFlow</div>
<div>
<br /></div>
<div>
<b>3. VRS (Data Plane) - Virtual Routing and Switching plugin</b> inside the Hypervisor. It´s based on OVS, and it´s responsible for L2/L3 forwarding, encapsulation.</div>
<div>
<br /></div>
<div>
On VRS you can define various VSC for redundancy and load balancing (one active and one standby), and each of them establishes an OpenFlow session using the Underlay network, not Management , using TCP port 6633 (SSL is optional).</div>
<div>
<br /></div>
<div>
VRS includes two main Nuage components:</div>
<div>
<br />
<ul>
<li>VRS Agent, that talks to VSC using OpenFlow. It's responsible for programming L2/L3 FIBs, and it replies to all ARP (no flooding). It also reports changes in VMs to the VSC. The forwarding table is pushed to VRS from VSC via OpenFlow. It has not only a view of all the IP and MAC addresses of the VMs being served by the local hypervisor, but also those which belong to the same domain (L2 and L3 segments), that is, all possible destinations of traffic for the VMs being served by that HV.</li>
<li>Open vSwitch (OVS), provides Switching and Routing components and Tunneling to forward the traffic.</li>
</ul>
</div>
<div>
<br /></div>
<div>
VRS supports a wide range of L2 and L3 encapsulation methods (VXLAN, VLAN, MPLSoGRE) so that it can communicate with a wide range of external network endpoints (other hypervisors, IP- or MPLS-based routers).</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s9.postimg.cc/x4cisj9u7/Screenshot_2017-06-20_20.26.33.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="417" data-original-width="800" height="207" src="https://s9.postimg.cc/x4cisj9u7/Screenshot_2017-06-20_20.26.33.png" width="400" /></a></div>
<div>
<div>
<br />
Let's get even deeper into the connection between the Control Plane and Data Plane, or VSC and VRS in Nuage Language. Nuage Networks uses Open Source components, such as libvert, OVS and OpenFlow. Nuage Networks makes use of the libvirt library in the VRS component that runs in Linux-based hypervisor environments (Xen and KVM) to get VM event notifications (new VM, start VM, stop VM, etc.). Libvirt is a package installed on the Hypervisor. Nuage also installs Nuage VRS. This enables the usage of User space tools:</div>
<div>
<br />
<ul>
<li>Virt-Manager: For GUI</li>
<li>Virsh: Commands (CLI)</li>
</ul>
</div>
<div>
<br /></div>
<div>
Before we continue, let's make sure we understand the basic concepts needed to understand the VRS and VGS (VRS-G included). Basically we need to understand:</div>
<div>
<br />
<ul>
<li>What is OVS (Open virtual Switch).</li>
<li>Difference between the Underlay and the Overlay.</li>
<li>What is VxLAN, what are VTEPs, and how it all works.</li>
</ul>
</div>
<div>
<br /></div>
<div>
<b>Open vSwitch (OVS)</b> is a major building block for Nuage SDN. It implements a L2 bridge includingu MAC learning. . OpenFlow is used to configure the vSwitch. It's used for Linux Networking and it's part of Linux Kernel, now used instead of Linux Bridge. OVS can be configured via CLI, OpenFlow or OVSDB management protocol. OVS doesn’t work like VMware VDS or Cisco 1000v. Instead, it only exists on each individual physical host, and it makes it easier for developers of virtualization/cloud management platforms to offer distributed vSwitch capabilities. In Nuage, OpenFlow is used to program the virtual switch within the hypervisor, with the vSwitch becoming the new edge of the datacenter network. The OVS becomes the access layer of the network. The access is where control policies are typically implemented: ACLs, QoS policies, monitoring (netflow, sflow), OVS has these features, and also provides an SDN programmatic interface (OpenFlow and OVSDB management).</div>
<div>
<br /></div>
<div>
The three main components of OVS are:</div>
<div>
<br />
<ul>
<li><b>ovsdb-server </b>is the configuration database which contains details about bridges, interfaces, tunnels, QoS, etc.</li>
<li><b>OVS kernel module </b>handles the data path, including packet header handling, table lookup and tunnel encapsulation and decapsulation. The first frame of a flow goes to ovs-vswitchd to make the forwarding decision; the following frames are then processed by the kernel.</li>
<li><b>ovs-vswitchd</b> matches the first frame for a “flow” action (L2 forwarding, mirroring, tunneling, QoS processing, ACL filtering, etc.) and caches these in the flow table in the kernel module.</li>
</ul>
</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s11.postimg.cc/pkuky8dw3/Screenshot_2017-06-17_19.22.48.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="476" data-original-width="800" height="237" src="https://s11.postimg.cc/pkuky8dw3/Screenshot_2017-06-17_19.22.48.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<div>
The Open vSwitch is configured by the “control cluster” through a combination of the following methods:</div>
<div>
<br />
<ul>
<li>SSH and the CLI can be used to manually configure the switch locally</li>
<li>The OVSDB management protocol is used to create switch instances, attach interfaces and define QoS and security policies.</li>
<li>OpenFlow is used to establish flow states and the forwarding tables for these flows</li>
<li>Netlink is the Linux communication API used between kernel and user space</li>
</ul>
</div>
<div>
<br /></div>
<div>
Open vSwitch can also be implemented on hardware switches, for example an SDN white box switch, as OVSDB management protocol is also implemented on some vendors’ switches.</div>
<div>
<br /></div>
<div>
Overlay Network: Virtual abstraction built on top of a Physical Network. There are Network-Centric overlays (VPLS, TRILL, Fabric Path) where hosts are not aware of the Overlay, and Host-Centric (VxLAN, NV-GRE, STT) where hosts help create the virtual tunnels.</div>
<div>
<br /></div>
<div>
<b>VxLAN</b>: You can check out my previous posts (go to Blog Map) for more details on how VxLAN Control Plane and Encapsulation take place. VXLAN has a 24 bit VXLAN identifier, which allows for 16 million different tenant IDs. The VXLAN UDP source port is set on the sending side with a special hashing function that allows for load balancing of traffic by ECMP (equal cost multiple path) in the datacenter network. Destination Port is 4789. On the data plane, each VTEP capable device needs to have a forwarding table with each possible destination MAC address within the same L2 domain and the hypervisor hosting it. The VNI identifies the L2 domain within the DC.</div>
<div>
<br /></div>
<div>
More and more server NIC cards support VXLAN offload functionality, which improves the encapsulation/decapsulation performance.</div>
<div>
<br /></div>
<div>
All VTEPs (Virtual Tunnel End Points) in the VxLAN Control Plane need at least the IP connectivity. VTEP needs to act as the default gateway for all the subnetworks that its hosted VMs belong to. In order to do this, VTEP will be assigned a MAC address and an IP address within each of such subnetworks. The combination of the IP and MAC addresses corresponding to a given VM is known as EVPN prefix. When a packet is sent by a VM to its default gateway, because its final destination is an IP address in a different subnetwork, the VTEP will look into its EVPN route table, swap the destination MAC address (presently pointing to the default gateway) to the MAC address of the VM intended to receive the packet, and send the frame to the VTEP hosting the destination VM using the corresponding VXLAN tunnel.</div>
<div>
<br /></div>
<div>
BGP EVPN is an Address Family that can include both, IP and MAC address for a given end point. Forwarding tables on each hypervisor contain information about all VMs in all subnets (each subnet corresponds to a different EVPN instance). VXLAN tunnels exist to reach these subnets on all the hypervisors. Backhaul VPLS brings optimization and enhanced scaling for the number of EVPN MAC addresses and tunnels. With this optimization, each VRS receives only complete forwarding information related to subnets (EVPNs) locally hosted on itself. Each VRS is still aware of every VM in remote subnets, the hypervisor hosting it and its IP address (but not its MAC address). Consequently, when a VM wants to communicate with another VM in a remote EVPN, the VRS (acting as the default gateway) only has to do a route-table lookup to identify which hypervisor is hosting the relevant IP address. This way, it can use the VXLAN tunnel indicated by the backhaul VPLS to forward the packet. There is no need in this case to find the corresponding VPLS and to do an additional L2 FDB lookup to determine the destination MAC address, as would happen if the subnet were not remote.</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s7.postimg.cc/dw2cu6j6z/Screenshot_2017-06-18_15.05.45.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="467" data-original-width="800" height="232" src="https://s7.postimg.cc/dw2cu6j6z/Screenshot_2017-06-18_15.05.45.png" width="400" /></a></div>
<div>
<br /></div>
<div>
VRS is in the Underlay Network, and OVS is in the Overlay Network. All Hypervisors need at lease one interface connected to the Underlay Network. You can also have the VTEP assigned to the ToR Switch instead of a Hypervisor, but the concepts don't change.</div>
<div>
<br /></div>
<div>
<a href="https://s7.postimg.cc/je5vr8jbv/Screenshot_2017-06-18_18.26.32.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="266" data-original-width="800" height="132" src="https://s7.postimg.cc/je5vr8jbv/Screenshot_2017-06-18_18.26.32.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<div>
VSG - Virtual Services Gateway allows the interconnection between Physical and Virtual domains. It basically translates VLAN to VxLAN (VxLAN towards Nuage Overlay, and VLAN to Legacy Infrastructure). There are two Nuage versions (physical and virtual), and a version for the "White Boxes":</div>
<div>
<br />
<ul>
<li>Software (VRS-G) which offers Network ports via Overlay (VxLAN) and access ports to the traditional network (VLAN)</li>
<li>Hardware, 7850 VSG is a 10/40G Switch providing VTEP GW functionality (VTEP in Hardware).</li>
<li>Hardware VTEP on a White Box</li>
</ul>
</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s11.postimg.cc/3nptdqdv7/Screenshot_2017-06-17_19.45.48.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="421" data-original-width="800" height="168" src="https://s11.postimg.cc/3nptdqdv7/Screenshot_2017-06-17_19.45.48.png" width="320" /></a></div>
<div>
<br /></div>
<div>
VSN is a Virtual Service Node, composed of VSC and a group of VRS. VSC is like a Control Plane, and VRS are like a Hypervisors. The VSN provides the network operator with a unified view of all the elements being handled by it, making HVs appear as line cards in a chassis when compared to a classic router. It provides a one-stop management and provisioning point for all the HVs under the VSN control.</div>
<div>
<br /></div>
<div>
<a href="https://s15.postimg.cc/i6csk4vzv/Screenshot_2017-06-17_19.03.37.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="489" data-original-width="800" height="243" src="https://s15.postimg.cc/i6csk4vzv/Screenshot_2017-06-17_19.03.37.png" width="400" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
<b>4. Don’t forget the Security: NFV and Service Chaining</b></div>
<div>
<br /></div>
<div>
Security Policies are defined at a Domain level, define From/To Zones and/or Subnets. It is important to understand the relative directions of security policies before implementing them. The easiest way to understand the directions is to imagine it from the OVS point of view. This would mean that the INGRESS would me traffic entering the OVS, and Egress - traffic going OUT of the OVS:</div>
<div>
Ingress refers to the direction of traffic flow from the VM towards the network (or the OVS component).</div>
<div>
Egress refers to the direction of traffic from the network (or the OVS component) towards the VM</div>
<div>
<br /></div>
<div>
Policies have priorities which allows defining the order. They can be Imported/Exported between the Domains, or to/from a File. Before you apply the policies:</div>
<div>
By default all INGRESS traffic is dropped (INGRESS means from VM to the OVS).</div>
<div>
By default all EGRESS traffic is accepted (from OVS to VM).</div>
<div>
<br /></div>
<div>
When defining the Security Policy, it's important to have in mind Nuage mode of operation, shown on the diagram below.</div>
</div>
<div>
<br /></div>
<div>
<a href="https://s1.postimg.cc/h31glq3q7/Screenshot_2017-06-18_12.37.36.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="451" data-original-width="800" height="180" src="https://s1.postimg.cc/h31glq3q7/Screenshot_2017-06-18_12.37.36.png" width="320" /></a></div>
<div>
<br /></div>
<div>
<div>
At the time of creation, a Policy Group Type is assigned to each Security Policy:</div>
<div>
Hardware, for hosts and bridge vPort hosted in Nuage VSG/VSA Gateways</div>
<div>
Software, VRS and VRS-G hosted vPort, including VM, host and bridge vPort</div>
<div>
<br /></div>
<div>
Important: When you do Stateless, you need ACLs for "returning" traffic. For stateful you just need one policy in one direction.</div>
<div>
<br /></div>
<div>
ACL Sandwich feature enables a network admin to define a supra-list that will drop specific traffic that should NEVER reach the VM. The end user who owns the domain instance can then combine ACL rules into ACLs defined on the domain instance level.</div>
<div>
<br /></div>
<div>
Logging can be able on ACL entry level.</div>
<div>
<br /></div>
<div>
<b>Service Chaining</b></div>
<div>
VSP provides so called Forwarding Policies to control the redirection of packets. This is what later enables Service Chaining. In my opinion, Nuage has the most elegant implementation of Service Chaining of all SDN products out there. All is implemented through flow-based redirection.</div>
<div>
<br /></div>
<div>
Nuage supports Physical and Virtual L4-7 Appliances/Cluster of Appliances as redirection targets, and it gives you the option of creating the Advanced Redirection Policies, where you're given the option to redirect only the traffic destined to a certain TCP/UDP port.</div>
</div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com5tag:blogger.com,1999:blog-6091645117172561542.post-15628273925223778152017-06-24T09:36:00.000+02:002017-07-02T05:22:09.401+02:00How to sell SDNThe most important thing about presenting SDN to a potential Customer, and about how you need to focus your Presentation, and I cannot stress this enough: your entire speech needs to be adapted to your audience.<br />
<br />
<b><span class="Apple-tab-span" style="white-space: pre;"> </span>1. Networking and Security Department</b><br />
<br />
What you need to know before you start planning the presentation:<br />
Before we get to the point, you need to understand that the Networking guys do not want SDN. Within the Networking department you will easily distinguish two types of engineers:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>- The ones who hate SDN, hate you for presenting it, and just want to continue doing things their own way.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>- The ones who understand that unless they understand and learn SDN, the System guys will choose the product, learn it, and take care of Networking themselves, making the Networking department obsolete. You should always direct to this group in your presentations.<br />
<br />
<b>What's the most positive thing SDN brings to the table?</b><br />
<br />
SDN is a concept of a Network that is Multi-Tenant, that has a single point of control of the entire Network, and most importantly - allows you to "consume" the Network using the APIs. This means that the Network department can give your Developers, or Cloud Admins, the tools and teach them to consume the Network. This way you avoid a usual delay that Networking department needs to configure Networking and Security for the new Apps and Services, and most importantly - the concept of Tenant allows them to use overlapping IPs, VLANs, Names, without ever being able to compromise the stability of your Network.<br />
<br />
What will they want to know? Here is a day to day of a Network Admin:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>- Something stops working.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>- In average it takes around 10 micro seconds before someone says "Hey, maybe it's a networking issue?"<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>- Regardless if your Network Admin has "more important stuff to do", he ends having to verify the entire Networking environment, because of a "issue in a production environment" and everything goes on top of the network.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>- The issue gets resolved. More often then not, Network Admins get no feedback about how they solved it.<br />
<br />
Network Admins just want no one to shout at them because something isn't working correctly.<br />
<br />
<a href="https://s21.postimg.org/96nkuux9z/IMG_4018.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="306" data-original-width="460" height="265" src="https://s21.postimg.org/96nkuux9z/IMG_4018.jpg" width="400" /></a><br />
<br />
<br />
This just means that the only thing the Network Admins will be demanding from your SDN solution is a set of easy-to-use Troubleshooting tools. Have this in mind when preparing your presentation.<br />
<br />
<br />
<b><span class="Apple-tab-span" style="white-space: pre;"> </span>1. Systems/Cloud Department</b><br />
What's the most positive thing SDN brings to the table?<br />
Networking department is handling so many "critical production issues" that they hardly have any time to provision the networking for new services. Even when they have time, they have to take so much care just not to break something in the network while configuring new stuff. In the world where it takes us seconds to bring up a new instance of VM or a Container, the current Network model just won't do. System guys need a way to simply provision the networking without writing an essay to the Networking department detailing why their request needs to be prioritized. This is why it will be really easy to make these guys understand (and probably love) any SDN solution you might be presenting.<br />
<br />
What will they want to know? This depends on the solution you're trying to position. Have in mind that these guys will love "graphical" solutions, such as VMware NSX and Nokia Nuage , and since they have a limited knowledge of Networking, it will be complicated to explain the advantages of the solution that handles Physical + Virtual Network, such as Cisco ACI and OpenDayLight.<br />
<br />
<br />
<b><span class="Apple-tab-span" style="white-space: pre;"> </span>3. Software Developers</b><br />
Developers have similar "needs" like the System guys, they need a way to simply provision and secure the communication flows. If you tell them that the solution you're presenting gives them the possibility to consume the Network using API calls - they're on board.<br />
<br />
<b><span class="Apple-tab-span" style="white-space: pre;"> </span>4. Mixed audience</b><br />
This is probably the most complex audience you can possibly have when talking about SDN, because each of the departments will understand the concept in a different manner. Be sure that you can handle the opened discussion, you have to be a true SDN Ninja to handle the "lost in translation" paradox that will occur. I strongly advice you to bring both, Networking/SDN and Systems Experts to a presentation of this type, and make sure that YOUR experts agree on what SDN is before you let them approach the client as a team.<br />
<br />
<div>
<br /></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com0tag:blogger.com,1999:blog-6091645117172561542.post-90260115821276394292017-06-02T13:18:00.002+02:002018-03-04T15:48:55.764+01:00What are Cisco Cloud Center (CliQr) and UCS Director, how to choose/integrate?Before we get into the details about each technology, and how you should choose which one best fits in your environment, I would strongly advise you to sit down and think about what exactly you need, what would be your ideal target environment. While doing this here are a few questions you need to ask yourself:<br />
<br />
<ul>
<li>What do I want to offer, IaaS, PaaS, SaaS, or a combination of these?</li>
<li>Do you want to automate the Application Deployment or Infrastructure Deployment?</li>
<li>Are you really ready for automation? I strongly believe that once you choose your Platforms, you should stick to it, because everything can be done in each of these… It's just that some are more suitable for certain tasks/ways of use then the others.</li>
</ul>
<div>
<br /></div>
<br />
USC Director is used for the Infrastructure Automation and Management (yes, management as well!). UCS has a huge Task Library for Infrastructure Elements such as Cisco Nexus and ACI, UCS, NetApp, EMC, vCenter, VMware vSAN etc.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s18.postimg.org/p1tj8eiw9/UCS_Director.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="483" data-original-width="800" height="241" src="https://s18.postimg.org/p1tj8eiw9/UCS_Director.png" width="400" /></a></div>
<br />
The main competitors of UCS Director are:<br />
<br />
<ul>
<li><b>vRealize Suite (Automation, Orchestration) by VMware</b>. I've seen very cool projects done with vRealize, but typically it's optimized for a mostly VMware environment.</li>
<li><b>Terraform by HashiCorp</b>. Linux geeks tend to love this one, as it only has a command mode, and you can deploy your infrastructure directly using the Code.</li>
<li><b>Ansible by RedHat</b>. You right your own Playbooks, they are human-readable. Very flexible.</li>
</ul>
<br />
<br />
<b>Why choose UCS Director?</b> It really depends on your environment and what you want to do. In my opinion it's a perfect fit when you want to include the automation of the physical infrastructure in your Workflow, and get the unified support by Cisco. Out of the Box UCS Director has bunch of Tasks already at your disposal (as you probably guessed, most Cisco products, such as ACI, Nexus, UCS etc. are already included). If you need to add tasks, there is a pretty nice community. Just check this one, it's a UCSD Workflow INDEX (UCSD Technical Content Index):<br />
https://communities.cisco.com/docs/DOC-56419<br />
<br />
<b>TIP</b>: If you are really interested in UCS Director I strongly advise you to build your own Lab and test it, before you make a purchase. Don’t trust that Power Point, the stakes are to high. There is a built in Evaluation License in a UCS Director, and you can download it as an OVA or VMDK from Cisco.<br />
<br />
<b>Cisco Cloud Center (ex Cliqr)</b> is a CMP (Cloud Management Platform). It is was a pretty pleasant surprise for me to see that Cisco is finally learning how to do Software products. In all fairness most of the original code comes from the company they acquired (Cliqr), but still… they also bought Insieme and turned it into ACI, and… well, you know how ACI GUI is.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s29.postimg.org/gv23b2grr/Cliqr_CMP.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="357" data-original-width="800" height="284" src="https://s29.postimg.org/gv23b2grr/Cliqr_CMP.png" width="640" /></a></div>
<br />
<br />
The main competitors of Cisco Cloud Center are:<br />
<br />
<ul>
<li><b>CloudForms by RedHat</b>. While CloudForms is more flexible, it doesn’t come with Libraries so you will need to do most coding yourself.</li>
<li><b>vRealize Suite</b> again, since it now supports Public Cloud.</li>
<li><b>Rightscale</b>, which purely follows SAAS model. You can not deploy Rightscale in your environment. Its already hosted somewhere, all you do is, login to it, add your cloud solution account and start managing it.</li>
<li>Others (<b>CloudBolt, Oracle</b> etc.).</li>
<li><b>Dell Multi-Cloud Manager </b>(don’t use this one, sorry @Dell).</li>
</ul>
<br />
<br />
<h3>
These sound similar, should I use UCS Director, Cloud Center, or both?</h3>
A short answer would be - UCS does Infrastructure, Cloud Center does Application. This does not mean that UCS Director couldn’t automate the Application deployment, or that Cloud Center cannot do the infrastructure. It means that both products are better suited doing what they were designed to do. Now, go back to the first paragraph and answer the questions. At this point you should have a clearer picture which is the right product for you.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s29.postimg.org/o09balcp3/Cliqr_Application_Modeling.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="563" data-original-width="800" height="281" src="https://s29.postimg.org/o09balcp3/Cliqr_Application_Modeling.png" width="400" /></a></div>
<br />
What if you need both, Application Deployment automation with Infrastructure modifications in accordance with the Application needs? In such case, you would use both, UCS Director as Day 1 product, and the Cloud Center for Application Deployment in Multi Cloud. On top of both these you would need an Orchestrator of Orchestrators. This is where you would place your Service Catalogue which would then use UCS Director and Cloud Center Northbound APIs to Automate your Application Deployment, doing the Application Tiers and Infrastructure deployments separately.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://s22.postimg.org/lysoc3bvl/ECS.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="345" data-original-width="800" height="172" src="https://s22.postimg.org/lysoc3bvl/ECS.png" width="400" /></a></div>
<br />
If you don’t want to build your own Service Catalogue Web, Cisco has a product of this type called PSC (Platform Service Catalogue). It's simple, but I'm not really sure how expensive it is… after all, it is Cisco.<br />
<div>
<br /></div>
Mats Cloudhttp://www.blogger.com/profile/18206886648747581607noreply@blogger.com3