Browse Category

VMware

Homelab: Compute

A little while ago, I wrote up some goals for the Homelab. The idea behind these goals was not so much to build a lab that compares in power to a normal DC, but to build something that can match in the way its configured. This is for my own continued education as well as for use in Demo’s of products to customers.

So lets take a look at the one of the areas mentioned in more specific terms. Compute. In my Goals post, I listed the following items as goals to look for. I must admit, I already had a product family in mind by the time I made these goals, just not a specific model chosen. Never the less, lets revisit that list.

Compute:

  • 1U rack height
  • 4 or more Cores
  • Dual NIC’s
  • 1-2 PCIe slots
  • Greater than 32GB RAM Max
  • IPMI (Dedicated or Shared NIC)

 

What Matters Most?:

In the case of my lab, it matters more to be able to match a configuration than it does to have the most powerful one ever. As an Engineer at heart and a Pre-Sales Systems Engineer, its important to work through configurations for accuracy to how customers deploy their infrastructure. More specifically, how customers deploy ESXi. Everything from the vmkernel IP’s to advanced settings on processor performance.No as most people would consider, I could have done this in a nested environment. Nested is a great idea when you are studying for test or wanting to do functional testing, but long term it has its limits. Nested was too underpowered and giant dual socket, 6-8 core servers were too power hungry! Also as a side note, this was going to be in my office and noise also needed to be taken into consideration quite a bit.

 

Core Count:

I decided that a 4-core machine with a single socket was plenty powerful. Take a look at all of the Intel NUC blog posts and you’ll see that they have what it takes in power, but are a little light on RAM. Spoiler alert: the only time i’ve hit almost 100% CPU was during the vRA deployment. Even now, I have one box at 40% and two at 20%. 4-Cores are more than enough, even without right sizing the VM’s down.

 

Network Connectivity:

As most labs go, 1G Networking is sufficient. Its hard to saturate a 1G switch in a homelab by most peoples standards. When I started looking at hardware, I looked at the prices of 10G networking and it put me off. At the time, 10G network switches would have put me back about $1500, and those only have 8-12 ports. Though the cost has come down since then, it was insane. My plan was to buy hosts with PCI slots available for 10G cards later. My major concern here was that I needed to make sure that any server purchased, had two physical NICs onboard. I’ll hand it to William Lam for building that into the NUC’s, but this needed to be clean, built in and ready to go with minimal firmware work.

Now at the same time I was looking, Supermicro had just announced their X10SDV line of motherboards. These are Embedded Xeon boards with 10G Ethernet or SFP+ built in. Let me repeat that really quick… 10G Networking built into the motherboard! That was potentially going to save me about $200 down the line, or in my case cost me about $200 more now vs later if I bought them. This lends towards my goal of upgradability. I could start with 1G network switches now and only need to swap the switches and cables later to upgrade. tempting. It wasn’t necessary but would definitely get me closer to “real world server config” as possible.

 

Memory:

Downside to the Intel NUC’s is the 32GB or RAM. RAM will always be the bottleneck in the DC and its no different in the homelab. I needed to make sure that any servers I run could handle more than 32GB. Its a common complaint amounts the vExperts and its one I wanted to avoid. When I started, I really liked the Shuttle PC’s. Their form factor has been well known for years and they have made some advances in max RAM capacity. Also to note is that most of those with higher limits also had dual NIC’s, so that was a plus. When the Supermicro boards came out though, they blew the competition out of the water. 128GB of RAM max capacity in 4 DIMM slots. Downside there is cost. 32GB DIMMs at the time cost about $250 per DIMM, ouch! Altogether still needed to make sure I wasn’t limited.

 

OOB Management:

In a previous role, I used both iDRAC and IPMI. I was leaning towards IPMI, only because to get something with iDRAC, I would have had to sacrifice  for noise levels and power consumption. That wasn’t going to happen and there are plenty of boards out with IPMI now. An added bonus to going with IPMI was all of the open source central management solutions out there. In the past I’ve used xCAT, developed by IBM/Lenovo engineers and made available as open source to their server users. It gave me a CLI for managing a whole datacenter’s worth of hardware and uploading firmware to hosts from a single point of management. Ideally, I would want to do the same here. If IPMI is shared with a network port or the board has a dedicated port, that didn’t matter. What matters was getting ISO’s to hosts via the network and not using any KVM equipment.

 

End Result?:

When I started looking into what I would use for compute, the X10SDV’s were still just a marketing promise. While working on a couple ideas for whitebox configs, Supermicro went from marketing to production. For a short time, I considered doing an open-air deployment and doing something like the Ikea Helmer Render Farm or something closer to how some larger DC’s place motherboard on rack trays (this was something that seemed particularly interesting to me).

In the end I went with the SYS-5018D-FN8T (or X10SDV-TP8F Motherboard). I went with this specifically for:

  • 35W TDP
    • This plays into low power and how quiet the server runs.
  • 6x Dedicated 1G NIC’s
    • If you were going to start with multiple 1G NIC’s to put off 10G networking as long as possible
  • 2x 10G SFP+ ports
    • BOOM! Favorite reason #1
  • 128GB RAM Max
    • Favorite reason #2
  • Small form factor
    • Comes in the same 10″ deep server chassis that I was already looking at for whitebox configs.
  • PCI Expandability
    • This is more for future use, looking at a Supermicro HCL storage controller.
  • Dedicated IPMI
    • also… Supermicro has Central Management tools in abundance. Depending on how you want to access or what you want to manage.

 

The server is a little more expensive than I was initially interested in, but meets my power and noise requirements. Noise and power were huge concerns for me, when originally looking at used servers on Ebay. I had to also consider adding a new line to the office, given that the main circuit breaker was full from when the house was built, it would be run from the main box outside. So the SYS-5018D-FN8T helps conserve power which really worked for me, but still provided upgradability that I knew I would need. I had started with two servers initially, and 64GB of RAM each (2x 32GB DIMMs, with an internal discount for Kingston). Then held out a little longer for a 3rd node for HA and potentially vSAN later on. All in all I love these servers. They are doing a great job and holding up really well.

When building the homelab, just remember to think about what you truly want out of it. I chose low power, cooling management over just functional testing, especially since I use this to show customers. Depending on your personal “business value” that could be different.

 

As a side note:

I attempted to purchase all of the same parts for 5018D-FN8T, and its not worth the time and effort to assemble over the small saving. Buy the completed system and only add the active cooler fan if you think you will put a little extra stress on the machines.

VMworld First Timers!

First off, I am still alive, just acclimating to my new role and the travel with VMware. To get back into the swing of things and as I get ready for some new things coming up, I thought this would be a good return to writing post.

So for those of you preparing to go to your first VMworld, CONGRATS! You are headed to an excellent conference with roughly 27,000 other people! Yes, let that sink in for half a second… twenty. seven. thousand. people. And that is if they didn’t get additional people to show up this year!

So, for you first timers, I know how you feel. I was you last year, and these are the tips that I got and learned myself that I want to share.

  • SHOES: Get a pair of shoes that are comfortable for walking… 10-15 thousands steps a day in. If you don’t have a step tracker, then you should get one. If you decide to buy a new pair of shoes, BREAK THEM IN FIRST! That was my mistake, I didn’t.
  • SESSIONS: You won’t make them all, it just doesn’t happen. You can try and plan the perfect schedule full of amazing sessions. You’ll want to be there for them all, and then, you’ll walk into the Solutions Exchange. You’ll walk out and realize you missed half the day. Don’t feel bad, it happens to everyone.
  • EVENTS: Lets call it what it is, parties. Call your partners, VAR’s and Vendors and see who has an event going on. This is a great chance to relax, meet people and just get off your aching feet for 30 minutes to an hour. (Because there are about 10 parties happening each night)
  • GOING ALONE?: I did this last year, and although you are surrounded by 27,000 people, there is no weirder way to feel alone. You don’t have to feel that way though. Have you ever met someone that you can talk shop with? Talk about technology and you just go on and on for an hour or two, before you realize time has passed? Yeah, you have 27,000 other people there just like that. This is my single biggest piece of advice. At times you will be walking with a large crowd in the same direction, looking for a table to sit at and see a single chair at a table of 5 or maybe sitting at an event. The best thing you can do at VMworld when you feel alone, or anti-social, just turn to your left or right and introduce yourself. You will be amazed at the people you meet. Heck, a couple times the people I met were VMworld presenters!
  • KEYNOTE: Although its great to attend the main keynote speech in the main hall, there are additional keynotes each day. Instead of going to the main hall, go to the community space. Sit with the bloggers, check out vBrownBag, talk with the VMware Engineers at the “Office of the CTO” Booth (VERY COOL FUTURE TECH!). Get out of the main room and the absolutely insane crowd. Oh and don’t expect to have great mobile data service during this time.
  • COMMUNITY: This is the real reason people go. At my last job, my coworker helped persuade my boss to send me to VMworld (THANK YOU!!) and this was his tip. Meeting with and networking in the community is so much more valuable than just attending the sessions (which are recorded and put on youtube). He was so very right. The people that I met at events, while walking towards a session and even in the community lounge I’ve continued to talk with via twitter and slack. Amazing people!

Thats it for now, but for those heading to their first VMworld, enjoy it! Its a great experience and I recommend hitting up all of the community based parties and events, those were my favorite.

Ramping up at VMware

I’m on week 3 at VMware, working my way through training and tasks designed to get me “Ramped Up” in the role. Its been incredible so far and for so many more reasons than I expected. First, let me say that I’m truly sorry I haven’t done the next Professional Growth post, I’m actually going to combine question 4 into post 3, and do a repost so look out for that. I’ve been a little busy but I promise to get back on track with those.

First, lets get this question out of the way.

“You went to VMware amidst the Dell Acquisition? Why?”

Yes, this was a major concern even before getting a call to do the first interview. There are so many articles and blog posts centered around VMware and Dell’s acquisition of EMC. Yes, there was a recent round of layoffs at VMware that was pretty significant. Ultimately, I felt comfortable with the stance VMware is showing and reached out to multiple contacts both inside and outside of VMware to get opinions.

I walked away from those conversations thinking that VMware, though part of the Federation, is very strong in its own right. This company still has new areas to grow and I’d like to be a part of that.

Alright, now that that is behind us, moving on.

Going to the “Dark Side”

I’ve heard this numerous times from my new coworkers and a few others. I understand that becoming a Pre-Sales System Engineer means that I am convincing companies that Product X is right for them, but its so much more than that. At the end of the first week, I questioned why that phrase is even used at all. By the end of the second, lets just say I don’t see it. This is an excellent opportunity and allows me to see and help so many people in so many different environments. I’m here to help validate, demo, show the value of these products and help solve problems. I don’t see a dark side to it at all.

First Impressions on my team and role

I wasn’t exactly sure what to expect going in. This is a vendor role, remember. Up until now, I’ve been a customer. One of the first things that struck me was all of the members willingness to help. People in different areas of the business have gone out of their way to get me slide decks and 1-on-1 meetings to discuss products, helping me get up and running.

My account reps took a good chunk of time to discuss and exchange knowledge and ideas, during a very busy part of their quarter, when they should be focusing on closing deals I obviously am not a part of. Above that, just being available far more than I expect them to be.

But, the best part of this actually happened around going to a customer’s business to do a demo. Now, as I mentioned earlier, I’ve been told “welcome to the Dark Side” plenty of times. This is part of sales, I support sales by doing the technical portion. It’s an understanding that I will convince companies that they need some software to solve their problem or reach their goal. What I wasn’t completely expecting was hearing the account rep and current systems engineer push on the idea of “we don’t sell shelf-ware”. The idea is that we don’t push products you don’t need to solve your current problem, or isn’t part of a soon to be project/goal.

That in itself makes this move all the sweeter. Why? Becuase it goes back to a core competency at my last job, a type of company culture if you will. At Voxeo/Aspect, they called it customer obsession. Doing what is right for the customer and seeing it through to the end. I’m reminded of that by this team and its amazing to see that idea being pushed, even in Pre-Sales.

Drinking from the firehose

Yeah, I have a lot of products to catch up on, but I’ve never been one to NOT want to learn a new enterprise tech. BRING. IT. ON.

Work | Life Balance

Coming from a customer background, I was presently surprised when my phone died the other night and I didn’t have to worry about an on-call rotation. But what really surprises me, is when my manager ends a friday call with “Alright, have a good weekend and remember, Family first… Work Second”. Work/Life balance is incredibly important here and as I agreed to this role, my one concern was how it might affect mine. I had spent a considerable amount of time working on this in my last job to improve it, to the best of my ability. While at VMware, its practically pushed in my favor by management.

Summary:

All in all, I already considered having VMware on my resume as a great career booster, but the perks keep rolling in. I don’t mean that in a way that suggests good discounts or benefits, which coincidentally are also great. The culture is good, the team is great and the role  will definitely cater professional growth beyond what I initially considered. I look forward the future that is at VMware. Now if I could only come up with a paper for VMworld… seriously, what should I talk about?!?!

This Method Requires Authentication

We were having some issues with one of our VCSA’s and creating or subscribing to Content Libraries. So here is our resolution.

Symptoms:

  • When creating a local Content Library, when clicking finish, it errors with: This Method Requires Authentication
  • When subscribing to another Content Library that has authentication disabled, after copying the json URL into the field and clicking next, it halts the view and states: This Method Requires Authentication
  • When attempting to download Support Bundles from the VAMI at https://<vCenter FQDN>:5480, Downloads timeout and fail

 

SSH into the VCSA and check the following log files:

/storage/log/vmware/vdcs/cls.log
/storage/log/vmware/vdcs/ovf.log
/storage/log/vmware/vdcs/ts.log

In cls.log, you will be looking for something like this:

cls.log
=========
2016-01-20T14:18:01.773Z | DEBUG    | unset-opId       | tomcat-http--39           | SsoOverRestVerifierUtil        | Trying to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/cls/resourcebundle
2016-01-20T14:18:01.800Z | ERROR    | unset-opId       | tomcat-http--39           | SamlTokenImpl                  | Signature validation failed
javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(Unknown Source)
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(Unknown Source)
        at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:653)
        at com.vmware.identity.token.impl.SamlTokenImpl.validate(SamlTokenImpl.java:535)
        at com.vmware.vim.sso.client.DefaultTokenFactory.parseToken(DefaultTokenFactory.java:46)
        at com.vmware.vim.sso.http.impl.AuthVerifierImpl.validateSamlToken(AuthVerifierImpl.java:77)
        at com.vmware.vim.sso.http.impl.AuthVerifierImpl.verifyToken(AuthVerifierImpl.java:66)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:183)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeader(SsoOverRestVerifierUtil.java:109)
        com.vmware.vcde.common.services.cm.servlet.SsoAuthenticatedFileStreamServlet.doGet(SsoAuthenticatedFileStreamServlet.java:103)
.
.
.
.
2016-01-20T14:18:01.801Z | ERROR    | unset-opId       | tomcat-http--39           | SsoOverRestVerifierUtil        | Failed to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/cls/resourcebundle
2016-01-20T14:18:01.801Z | ERROR    | unset-opId       | tomcat-http--39           | SsoAuthenticatedFileStreamServlet | doGet: SSO verification failed for client <vCenter IP Address>
com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil$SsoAuthException: com.vmware.vim.sso.http.AuthException: The SAML token is invalid!
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:194)

 

In ovf.log, you are looking for:

ovf.log
-------
2016-01-20T14:18:01.792Z | DEBUG    | unset-opId       | tomcat-http--23           | SsoOverRestVerifierUtil        | Trying to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/ovf/resourcebundle
2016-01-20T14:18:01.804Z | ERROR    | unset-opId       | tomcat-http--23           | SamlTokenImpl                  | Signature validation failed
javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(Unknown Source)
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(Unknown Source)
        at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:653)

        
        
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--23           | SsoOverRestVerifierUtil        | Failed to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/ovf/resourcebundle
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--23           | SsoAuthenticatedFileStreamServlet | doGet: SSO verification failed for client <vCenter IP Address>
com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil$SsoAuthException: com.vmware.vim.sso.http.AuthException: The SAML token is invalid!
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:194)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeader(SsoOverRestVerifierUtil.java:109)
        at com.vmware.vcde.common.services.cm.servlet.SsoAuthenticatedFileStreamServlet.doGet(SsoAuthenticatedFileStreamServlet.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at com.vmware.vcde.common.services.cm.servlet.DispatcherServlet.service(DispatcherServlet.java:53)

 

In ts.log, you are looking for:

Ts.log
---------
2016-01-20T14:18:01.792Z | DEBUG    | unset-opId       | tomcat-http--14           | SsoAuthenticatedFileStreamServlet | doGet: Entering (/ts/resourcebundle)
2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--14           | SamlTokenImpl                  | Signature validation failed
javax.xml.crypto.dsig.XMLSignatureException: the keyselector did not find a validation key
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature$DOMSignatureValue.validate(Unknown Source)
        at org.jcp.xml.dsig.internal.dom.DOMXMLSignature.validate(Unknown Source)
        at com.vmware.identity.token.impl.SamlTokenImpl.validateSignature(SamlTokenImpl.java:653)

2016-01-20T14:18:01.805Z | ERROR    | unset-opId       | tomcat-http--14           | SsoOverRestVerifierUtil        | Failed to verify request signature using following; host:<vCenter FQDN>, port: 443, uri:/ts/resourcebundle
2016-01-20T14:18:01.806Z | ERROR    | unset-opId       | tomcat-http--14           | SsoAuthenticatedFileStreamServlet | doGet: SSO verification failed for client <vCenter IP Address>
com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil$SsoAuthException: com.vmware.vim.sso.http.AuthException: The SAML token is invalid!
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeaderImpl(SsoOverRestVerifierUtil.java:194)
        at com.vmware.cis.services.common.sso.SsoOverRestVerifierUtil.verifySecurityHeader(SsoOverRestVerifierUtil.java:109)
        at com.vmware.vcde.common.services.cm.servlet.SsoAuthenticatedFileStreamServlet.doGet(SsoAuthenticatedFileStreamServlet.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

 

Cause:

According to VMware support, these log entries show no security context for the user. Without that Security content the user cannot perform actions on the content library.

 

Resolution:

We found the signing cert and its root CA used by SSO from vmware-identity-sts.log and took out the ssoserverSign and the root certificate and added them to the CA to TRUSTED_ROOTS using the below mentioned vets command.

/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store TRUSTED_ROOTS --alias roo51 --cert 51root.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry create --store TRUSTED_ROOTS --alias roo52 --cert 52root.crt

 

Then restart all services. Run the following commands. (No there is not a –restart or –reset, use both commands).

service-control --stop --all
service-control --start --all

Thats all for today folks. Hope this helped!

Professional Growth: Part 2 & 3 – Base Requirements and Upgrades

Continuing the series on professional growth, which started with my last post: Part 1 – Inspirational Beginnings. I started off asking “How did you get started and how long have you been in the game?” I got some interesting results and this next post I asked the question: “What traits, methods, knowledge or experiences did you find crucial to your growth?” Also updated with question 3, “What has changed since then, what’s new to be done?” Keep Reading

Answering My Own Questions

This week I reached out to a bunch of people in my followers list with a small list of questions. The purpose of this is to gauge what people in different roles, years of experience and geographies had to say about growth in the technology field. I’ve received some of the responses already and I have to say, I’m loving them. I can’t wait to do a write up. Just forgive me for the length, because there will be SO MUCH that I want to share from them. Keep Reading

Holy Crap.. this Community

One of the things about working in technology is how much people tend to share. For as long as there have been forums, chats and channels, there has been some form of community. Growing up, it was neat to build a computer and when I had an issue, I knew that someone somewhere had the same exact issue, using the same exact hardware. That person would likely have asked somewhere and I’d look for that thread.

Moving forward, as I was taking programming classes and working on java, C and later objective-c, I started going to the Orlando iOS Developers Meetup. I thought it was amazing to see a group of people getting together to talk about the tech. So as I progressed and I started working on virtualization, there was this mountain of information available.

Not all areas within technology are the same, some hold information close to their chest, so when I needed information on VMware, it was shocking how much was available. But its not just bloggers, the sheer volume of tweets dedicated to this is outrageous. Then, there is VMUG. I’ve been through and seen a couple communities, but the support and help surrounding virtualization (not just VMware) is incredible. The VMware User Groups are local groups, lead by volunteers who are also customers. Specifically, a leader CANNOT work for VMware.

So, if you’ve been a part of this community for awhile, thank you! If you are just joining, strap in, there is a LOT of content and the list grows hourly. I’m looking forward to 2016, to see what this community will do, and to see more users join in and sharing their knowledge.

ESXi 5.5 and older images in Update Manager 6

After recently upgrading from VCSA 5.5 Update 3 to VCSA 6.0 Update 1, we wanted to continue upgrading hosts to 5.5 Update 3 because we were waiting on Lenovo to release 6.0 Update 1a custom ISO’s.

In uploading the 5.5 U3a ISO’s to Update manager, both through the web-client and through the fat-client, we would get the following error:

Failed to import data. The uploaded upgrade package cannot be used with VMware vSphere Update Manager

After reaching out to VMware Support, we finally got this response:

This is to let you know that with With Update Manager 6.0 you can upgrade hosts that are running ESXi 5.x to ESXi 6.0. Host upgrades to ESXi 5.0, ESXi 5.1 or ESXi 5.5 are not supported.

So there it is, Update Manager only supports 6.0 and up images. Well, Lenovo’s ESXi 6.0 Update 1 target release date is December 22, 2015.