Archive for the ‘System Administration’ Category

IE 11 and CloudForms Certificate Issues

May 16th, 2016 No comments

We were receiving an error when using IE 11 against a CloudForms appliance: “Error: Name is required”. It appears to be due to IE’s handling of Self-Signed certificates.

Since we’re using IPA as the authentication source for CloudForms, I figured we’d just use that to generate certs for the appliance and then we can trust the CA from IPA. Here’s a gist of the shell script I whipped up to do that. It should be run from the CF Appliance that’s already joined to the IPA realm.

Packaging VMware VDDK 6.0.x RPM for CloudForms

April 27th, 2016 No comments


Here’s a SPEC file I came up with to package the VMware VDDK as an RPM for CloudForms/ManageIQ appliances. VDDK 5.x came with an install script, but 6.x does not. It’s probably better packaged as an RPM anyway. This is based off of the instructions available at (for paying Red Hat customers).

The latest versions will be available at


This is packaged as a nosrc.rpm since VMware-vix-disklib can’t be distributed. To build, run the following:

yum install rpmdevtools

Download the VMware-vix-disklib-6.0.2-3566099.x86_64.tar.gz to ~/rpmbuild/SOURCES
Download the vmware-vix-disklib.spec to ~/rpmbuild/SPECS

rpmbuild -ba ~/rpmbuild/SPECS/vmware-vix-disklib.spec

SPEC file

Name:           vmware-vix-disklib
Version:        6.0.2
Release:        1%{?dist}
Summary:        The Virtual Disk Development Kit (VDDK) is a collection of C libraries, code samples, utilities, and documentation to help you create or access VMware virtual disk storage.
License:        Proprietary
Source0:        VMware-vix-disklib-6.0.2-3566099.x86_64.tar.gz
NoSource:	0
BuildRequires:  coreutils
The Virtual Disk Development Kit (VDDK) is a collection of C libraries, code samples, utilities, and documentation to help you create or access VMware virtual disk storage. The kit includes:
* The Virtual Disk and Disk Mount libraries, sets of C function calls to manipulate virtual disk files. C++ code samples that you can build with either Visual Studio or the GNU C compiler
* Documentation about the VDDK libraries and the command-line utilities
* The Disk Mount utility to access files and file systems in offline virtual disks on Windows or Linux guest virtual machines
* The Virtual Disk Manager utility to manipulate offline virtual disk on Windows or Linux (clone, create, relocate, rename, grow, shrink, or defragment)
%setup -n %{name}-distrib
%__mkdir_p %{buildroot}/usr/lib/vmware-vix-disklib
%__cp -r bin64 include lib64 %{buildroot}/usr/lib/%{name}
%__ln_s /usr/lib/vmware-vix-disklib/lib64/ %{buildroot}/usr/lib/
%__ln_s /usr/lib/vmware-vix-disklib/lib64/ %{buildroot}/usr/lib/
%doc doc/*
* Wed Apr 27 2016 Matt Hyclak <> 6.0.2-1
Initial Build

Delaying VM Deletion during Retirement in CFME 5.4

September 29th, 2015 No comments

At $DAYJOB we would like to be able to “un-retire” a VM, especially in those cases where a customer retires it and “didn’t know that’s what it would do” or similar. To meet these requirements, I’ve introduced a delay in processing the full retirement by overriding the default pre_retirement.rb method. Here’s the gist of things:

# Description: This method powers-off the VM on the provider and waits for Retirement Delay to pass before continuing
require 'miq_dev_util'
@logger =$evm, 'pre_retirement')
vm = $evm.root['vm']
customer_id = $evm.instantiate('/Configuration/Methods/get_customer_identifier')['return_value']
retirement_delay = $evm.instantiate("/Infrastructure/VM/Retirement/Configuration/RetirementDelay/#{customer_id}")['retirement_delay'].gsub(' ', '.')
#TODO: Figure out a way to ensure retirement_delay is valid
@logger.log('info', "Found Customer #{customer_id} with retirement delay #{retirement_delay}")
# Power off the VM
unless vm.nil? || vm.attributes['power_state'] == 'off'
  ems = vm.ext_management_system
  @logger.log('info', "Powering Off VM <#{}> in provider <#{ems.try(:name)}>")
# If we don't have a delay, continue on
if retirement_delay.nil?
  @logger.log('info', "No retirement delay requested. Continuing with the process")
# Otherwise process the delay
  if vm.retirement_state == 'retiring'
    @logger.log('info', "Delaying #{retirement_delay} before finishing retirement for #{}")
    # The VM isn't really retired here, but for display purposes lets see what happens
    # The retirement date isn't set, so that may be a way to differentiate if we need to
    vm['retired'] = true # Because exposing the retired= method is too hard?
    vm.retirement_state = 'delaying'
    $evm.root['ae_result']         = 'retry'
    $evm.root['ae_retry_interval'] = "#{retirement_delay}"
  elsif vm.retirement_state == 'delaying'
    @logger.log('info', "Retirement delay reached. Continuing with the process.")
    vm['retired'] = false # Have to set this back so finish_retirement doesn't bomb later.
    vm.retirement_state = 'retiring'

Some items of note:

  1. I’m using the miq_dev_util module written by my colleague that provides a couple of nice utility functions. In this case I’m only using the logging, so swapping that out would be no big deal.
  2. The VM gets “retired” very early on so that the VM shows up with an ‘R’ for an icon and it is obvious it’s been retired – even though it hasn’t been deleted. During this time, the Retirement Status is set to “Delaying”.
  3. We take advantage of the retry capabilities of the state machine to perform the delay for us.
  4. customer_identifier is something internal to us. You can use whatever key you like to search for the appropriate instance of RetirementDelay. Eventually, retirement_delay should contain a ruby-valid time statement, such as ’60.minutes’ or ‘7.days’

Hope you find this useful!

Deploying ESXi with Satellite 6

June 11th, 2015 2 comments

Deploy the files to the Satellite server

  • Download the latest ISO from VMware (e.g. VMware-ESXi-5.5U2-RollupISO2.iso).
  • Transfer to the satellite server
  • Run some code
# mount -o loop VMware-ESXi-5.5U2-RollupISO2.iso /mnt
# mkdir /var/lib/tftpboot/boot/esxi55u2
# cd /mnt
# cp -a * /var/lib/tftpboot/boot/esxi55u2
# cd /var/lib/tftpboot/boot/esxi55u2
# sed -i 's#/#/boot/esxi55u2/#g' boot.cfg
# cd ..
# restorecon -R esxi55u2

Create the Operating System inside Satellite

Hosts -> Operating Systems

  • Click New Operating System
    • Operating System
      • Name: ESXi
      • Major version: 5
      • Minor version: 5
      • Description: ESXi 5.5U2
      • OS Family: Red Hat
      • Arch: x86_64
    • Partition Table
      • Kickstart Default
    • Installation Media
      • Any Mirror – Can create a esxi mirror if desired.

Hosts -> Provisioning Templates

  • Click New Template
    • Provisioning Template
      • Name: ESXi OCP PXELinux
        SERIAL 0 115200n8
        DEFAULT esxi5.serial
        PROMPT 0
        MENU TITLE PXE Boot
        LABEL esxi5.serial
             MENU LABEL ^4) ESXi55_Serial
             KERNEL boot/esxi55u2/mboot.c32
             APPEND -c boot/esxi55u2/boot.cfg text com1_Port=0x3f8 gdbPort=none logPort=none tty2Port=com1 ks=<%=foreman_url("provision")%>; ignoreHeadless="True"
        LABEL hddboot
        LOCALBOOT 0x80
        MENU LABEL ^Boot from local disk
    • Type: PXELinux
    • Association:
    • ESXi 5.5
  • Click New Template
    • Provisioning Template
      • Name: ESXi OCP Kickstart
        install --firstdisk --overwritevmfs
        rootpw --iscrypted <%= root_pass %>
        network --bootproto=dhcp
        %post --interpreter=busybox --ignorefailure=true
        # Add temporary DNS resolution so the foreman call works
        echo "nameserver <%= @host.subnet.dns_primary %>" >> /etc/resolv.conf
        echo "nameserver <%= @host.subnet.dns_secondary %>" >> /etc/resolv.conf
        wget -O /dev/null <%= foreman_url %>
        echo "Done with Foreman call"
        esxcfg-advcfg -k none gdbPort
        esxcfg-advcfg -k none logPort
        esxcfg-advcfg -k com1 tty2Port
        #script to set first boot options
        %firstboot --interpreter=busybox
        # enable VHV (Virtual Hardware Virtualization to run nested 64bit Guests + Hyper-V VM)
        grep -i "vhv.enable" /etc/vmware/config || echo "vhv.enable = \"TRUE\"" >> /etc/vmware/config
        # enable & start remote ESXi Shell (SSH)
        vim-cmd hostsvc/enable_ssh
        vim-cmd hostsvc/start_ssh
        # enable & start ESXi Shell (TSM)
        vim-cmd hostsvc/enable_esx_shell
        vim-cmd hostsvc/start_esx_shell
        # supress ESXi Shell shell warning
        esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1
        # ESXi Shell interactive idle time logout
        esxcli system settings advanced set -o /UserVars/ESXiShellInteractiveTimeOut -i 3600
        # Disable IPv6 for VMkernel interfaces
        esxcli system module parameters set -m tcpip3 -p ipv6=0
        # enable firewall
        esxcli network firewall set --default-action false --enabled yes
        # services to enable by default
        FIREWALL_SERVICES="syslog sshClient ntpClient updateManager httpClient netdump"
        esxcli network firewall ruleset set --ruleset-id ${SERVICE} --enabled yes
        # enter maintenance mode
        esxcli system maintenanceMode set -e true
        # Needed for configuration changes that could not be performed in esxcli
        esxcfg-advcfg -k none gdbPort
        esxcfg-advcfg -k none logPort
        esxcfg-advcfg -k com1 tty2Port
        esxcli system shutdown reboot -d 60 -r "rebooting after host configurations"
      • Type: provision
      • Association:
        • ESXi 5.5

Hosts -> Operating Systems

  • Select ESXi 5.5
    • Templates
      • provision: ESXi OCP Kickstart
      • PXELinux: ESXi OCP PXELinux



Pasteboard (clipboard) utilities in OSX

May 20th, 2015 No comments

I came across the pbcopy and pbpaste utilities today. It was handy, so I thought I’d post something mostly so I don’t forget. With file redirection, it becomes handy for things like pasting your public key into a website like github.

pbcopy < ~/.ssh/
Categories: System Administration Tags: ,

CloudForms VMware Tools Upgrade method

May 14th, 2015 No comments

I wrote a method a while back to allow me to upgrade VMware tools from within the CloudForms interface. I thought I would share it. I usually create a VM button to call the method, but it could probably be used elsewhere with some tweaking.

For the button, create it with System/Process/Request, Message create and Request upgrade_vmware_tools.

Screen Shot 2015-05-14 at 10.31.33 AM

Next, create an instance under /System/Process/Request called upgrade_vmware_tools that has a relationship field pointing to the location of your method. In my case, I chose /Infrastructure/VM/Operations/Methods/upgrade_vmware_tools

Screen Shot 2015-05-14 at 10.32.56 AM

Finally, you can create your instance and method.

Screen Shot 2015-05-14 at 10.33.32 AM

Here’s the code for what I wrote:

# CFME Automate Method: upgrade_vmware_tools
# Inputs: $evm.root['vm']
  # Method for logging
  def log(level, message)
    @method = 'upgrade_vmware_tools'
    $evm.log(level, "#{@method} - #{message}")
  def dump_attributes(my_object, my_object_name)
    $evm.log(:info, "Begin #{my_object_name}.attributes")
    my_object.attributes.sort.each { |k, v| $evm.log(:info, "#{my_object_name} Attribute - #{k}: #{v}")}
    $evm.log(:info, "End #{my_object_name}.attributes")
    $evm.log(:info, "")
  def dump_associations(my_object, my_object_name)
    $evm.log(:info, "Begin #{my_object_name}.associations")
    my_object.associations.sort.each { |a| $evm.log(:info, "#{my_object_name} Association - #{a}")}
    $evm.log(:info, "End #{my_object_name}.associations")
    $evm.log(:info, "")
  def dump_virtual_columns(my_object, my_object_name)
    $evm.log(:info, "Begin #{my_object_name}.virtual_columns")
    my_object.virtual_column_names.sort.each { |vcn| $evm.log(:info, "#{my_object_name} Virtual Column - #{vcn}")}
    $evm.log(:info, "End #{my_object_name}.virtual_columns")
    $evm.log(:info, "")
  log(:info, "CFME Automate Method Started")
  # Log information 
  #dump_attributes($evm.root, "$evm.root")
  #dump_associations($evm.root['vm'], "vm")
  #dump_virtual_columns($evm.root, "$evm.root")
  def find_vm(dc, name)
    vm = {}
    dc.datastoreFolder.childEntity.collect do |datastore|
      vm[:instance] = datastore.vm.find { |x| == name }
      if vm[:instance]
        vm[:datastore] =
  def upg_tools(vm)
    if vm[:instance][:guest][:guestFamily] == 'windowsGuest'
      instopts = '/s /v "/qn REBOOT=ReallySuppress"'
      instopts = nil
    $evm.log(:info, "Upgrading VMware Tools on #{vm[:instance][:name]}")
    upgtask = vm[:instance].UpgradeTools_Task(:installerOptions => instopts).wait_for_completion
  require 'rbvmomi'
  VIM = RbVmomi::VIM
  rvm  = $evm.root['vm']
  ems = rvm.ext_management_system()
  credentials = { :host => ems['hostname'], :user => ems.authentication_userid(), :password => ems.authentication_password(), :insecure => true }
  vim = VIM.connect credentials
  dc = vim.serviceInstance.find_datacenter
  vm = find_vm(dc,
  rc = upg_tools(vm)
  # Exit method
  log(:info, "CFME Automate Method Ended")
  exit MIQ_OK
  # Ruby rescue 
rescue => err
  log(:error, "[#{err}]\n#{err.backtrace.join("\n")}")
  exit MIQ_ABORT

Using the CloudForms SOAP API for daily provision reports

May 7th, 2015 No comments

For several reasons, we are currently unable to integrate directly from CloudForms to our CMDB. The current workaround is to send a CSV with the pertinent information to the group responsible for creating CIs in the CMDB. We implemented this originally as part of the Automation workflow – but after talking with this group, they indicated that the volume of individual CSV files would be overwhelming.

Read more…

Quick Script to join content hosts to Satellite 6

April 23rd, 2015 No comments

I wrote this for $DAYJOB to move machines registered with RHN or CentOS machines not registered to anything over to our Satellite 6 server.

I presume you have created activation keys titled organization-${DISTRO}${MAJ_REL}-development for all combinations of DISTRO (rhel, centos) and MAJ_REL (6, 7).

# Prepare to register with Satellite
# Make sure lsb_release is installed  
rpm -q redhat-lsb-core 2>&1 >/dev/null
if [ $? -ne 0 ]; then
  yum -y install redhat-lsb-core
# Which version are we running
RELEASE=`lsb_release -r -s`
# Are we on CentOS or RHEL?
if [ -f /etc/centos-release ]; then
  # Get the copr release of RHSM for CentOS
  if [ ! -f /etc/yum.repos.d/dgoodwin-subscription-manager.repo ]; then
    wget -O /etc/yum.repos.d/dgoodwin-subscription-manager.repo${MAJ_REL}/dgoodwin-subscription-manager-epel-${MAJ_REL}.repo
# Install the certificates - will pull in subscription-manager
yum -y install
# Install additional useful packages
yum -y install katello-agent
/sbin/service goferd start
# Register with Satellite
subscription-manager register --org="Example" --activationkey="example-${DISTRO}${MAJ_REL}-development"

Setting PAM requirements for password complexity and lifecycle

December 3rd, 2014 No comments

I had a fairly strict requirement at $work for a system that had the following password security requirements:

  • Minimum of 8 characters
  • Minimum of 2 different complexity classes
  • Password expires after 90 days
  • No reuse of the last 5 passwords AND no more than 1 password change per day
  • Temporary lockout after 5 failed attempts in 15 minutes, lockout to last for 15 minutes

Here’s how I solved it:

# cd /etc/pam.d
# cp system-auth-ac system-auth.local
# cp password-auth-ac password-auth.local

Modify both system-auth.local and password-auth.local to include pam_faillock and pam_cracklib. You should end up with something like:

auth required
auth sufficient nullok try_first_pass
auth [default=die] authfail audit deny=5 unlock_time=900 even_deny_root
auth sufficient authfail audit deny=5 unlock_time=900 even_deny_root
auth requisite uid >= 500 quiet
auth required

account required
account sufficient
account sufficient uid < 500 quiet account required password requisite try_first_pass retry=3 type= minlen=8 minclass=2 enforce_for_root password sufficient sha512 shadow nullok try_first_pass use_authtok remember=5 password required

Link the new files into place (to avoid authconfig from overwriting them):

# ln -sf system-auth system-auth.local
# ln -sf password-auth password-auth.local

Edit /etc/login.defs and change:


Apply immediately to root:

# chage -M 90 root
# chage -m 1 root

Categories: System Administration Tags:

Puppetmaster cert change

August 6th, 2014 No comments

Some day I will need this. Hat tip to Nan Liu.

From the puppet-users mailing list

Backup your puppet master ssl directory, so you can just retry if something didn’t go as planned.

# note all certificate alt names of the existing puppet master cert:
puppet cert -la | grep oldmaster
(alt names "DNS:puppet", "DNS:puppet-master", "DNS:puppet.mgmt", )
# remove your old puppet master cert.
puppet cert -c oldmaster
# search the ssl dir and it should not have any files with the oldmaster certname
# generate new master cert (same name as old one, but accept new_hostname in dns_alt_names):
puppet cert -g oldmaster --dns_alt_names=new_hostname,puppet,puppet-master,puppet.mgmt
# you may need to copy the files to some locations if you found files not removed after the cert clean step

At this point you can add a host entry on one of your agents and test via:
puppet agent -t –server new_hostname –noop

You should not have to touch any client cert, that’s only necessary if you need to change your CA cert which is a pain when it expires.



Categories: System Administration Tags: