I’ve recently run into a problem with 7-STABLE on VMware ESXi 3.5u4. With a recent change my VM shuts off shortly after probing the LSI (mpt) disk controller. The same behavior started occurring over the summer in HEAD and the quick workaround is to change the VM’s disk controller type from LSI to BusLogic. Lately I have some time to poke people about this issue so I figured I would. The problem is getting as much as I can while booting and having some usable boot messages for someone to look at. This would usually be accomplished by redirecting console output to a serial port on the problem machine and hooking up a cross over cable between it and another box. I haven’t done this on VMware before though so I had to do a little googling and it’s pretty simple.
On the FreeBSD side the following needs to be added to /boot/loader.conf:
console=”vidconsole,comconsole”
This will redirect the console to both the video display and a serial port. Once that is done shutdown the VM so the serial port can be added and configured.
With the crashing VM turned off go to “Edit Settings”:
Click the “Add” button to add a serial port to the VM:
For the serial port output select “Connect to named pipe”:
Finally configure the pipe:
The name of the pipe should be a file location on the VM host machine, not the guest. The near end is “Server” since I want to see the output from this VM and the far end will be another virtual machine. For the VM I’ll be connecting from to view the console output I would do the same but near end would be “Client”.
Once all this is done, from the second working VM launch cu(1).
# cu -l cuad0
After that boot the crashing VM and kernel messages should appear on the second VM. That’s all it takes to setup a serial connection between two FreeBSD VMs on VMware.
Update: Images fixed.
I ended up reinstalling ESXi 3.5u3 recently since there were no running needed VMs on the box and I wanted to play with a few changes I made to the installer image. I have a few steps I do after installation which seem common enough after doing some googling so I thought I just write up how to do them in one spot.
Enable SSH
You can do a lot of management from the VIC or RCLI but some days I just feel like logging onto the console and getting a shell. You can get to it from the console but I don’t like getting off the couch. SSH isn’t enabled by default but it is available.
Access the console on the box by hitting CTRL+F1, type “unsupported” (it doesn’t echo), and then enter the root password.
1) Edit the inetd .conf file:
# vi /etc/inetd.conf
2)Uncomment the SSH line:
:%s/#ssh/ssh/
3)SIGHUP inetd:
# ps | grep inetd | cut-f 1 -d ‘ ‘
Keep in mind that a user created through the VIC will not be able to ssh in since their shell is set to /bin/false and their home directory is non-existent. You can’t enable shell access using `vicfg-user` from the RCLI package so this needs to be done from console. The easiest thing to do is add the user via the VIC or `vicfg-user` and then edit /etc/passwd to change the user’s home directory to “/” and their shell to “/bin/ash”. (`usermod` can change the user’s shell but not thier home directory.)
Setup an SSL certificate
After installing I rename the box which means it’s time to get a new SSL cert to reflect that. You can run `/sbin/create_certificates` to create a self-signed certificate or you can create a certificate and have it signed by a CA. I have a CA setup here at home and signed the new key with that. I also created the key on that box as well. (You can just create an /usr/local/ssl/openssl.cnf file which is missing based on the `create_certificates` script if you want to create the key and csr on the ESXi host.) File paths of course differ depending on the OS you use to generate the key.
1) Generate the “rui.key” file:
# openssl genrsa -out /etc/ssl/private/rui.key 1024
2) Create the certificate signing request:
# openssl req -new -text -key /etc/ssl/private/rui.key -out /etc/ssl/rui.csr
3) Sign with your own CA. (Or send the file off to your CA.):
# openssl x509 -req -text -days 7000 -in /etc/ssl/rui.csr -CA /etc/ssl/CA.crt \
-CAkey /etc/ssl/private/CA.key -out /etc/ssl/rui.crt
4) Set proper file permissions:
# chmod 0400 /etc/ssl/private/rui.keychmod 0444 /etc/ssl/rui.crt
5) Copy the files over to the ESXi host and delete the ones just created:
# scp /etc/ssl/rui.crt /etc/ssl/private/rui.key esxihost:/etc/vmware/ssl
# rm /etc/ssl/rui.crt /etc/ssl/rui.csr /etc/ssl/private/rui.key
Reboot the ESXi host. (Not sure what specific service needs to be restarted.)
Send hostname by DHCP client
The ESXi DHCP client does not send the hostname but it can. I searched through the system to see where the dhcp client was started but couldn’t find it. I have a feeling it’s in the _vmkctl python module but couldn’t find the source for it to confirm this. Instead I just use a script that I snagged off the VMware forums and drop it into /etc/rc.local.d. The host ends up DHCPing twice but whatever.
1) Extract the oem.tgz:
# mkdir /tmp/oem/
# tar xzf /bootbank/oem.tgz -C /tmp/oemmkdir -p /tmp/oem/etc/rc.local.d
2) Create /tmp/oem/etc/rc.local.d/send-hostname:
#! /bin/ash
if [ -r /var/run/dhcp-vmk0.pid ]; then
kill $(cat /var/run/dhcp-vmk0.pid)
/bin/busybox udhcpc -i vmk0 -p /var/run/dhcp-vmk0.pid --retries=20 -H $(hostname | cut -d . -f 1)
fi
3) Make the script executable and then tar it up.
# chmod +x /tmp/oem/etc/rc.local.d/send-hostname
# cd /tmp/oem
# tar zcvf /bootbank/oem.tgz *
After that just reboot or copy the send-hostname script over to /etc/rc.local.d and run it.
These are the three things I usually do after installing ESXi and hopefully this will save a few people some extra googling. The next thing I’m trying to figure out is how to build applications for the service console. Specifically, I’d like to get pam_krb5 and nss_ldap working on the service console.
I’m running an x86_64 install for Fedora right now on my laptop and want to use the RCLI tools which fail to install on x86_64 (amd64) OSes.
[root@tomcat vmware-rcli-distrib]# ./vmware-install.pl
This version of "VMware VI Remote Command Line Tools" is incompatible with this
operating system. Please install the "x86_64" version of this program instead.
I found a patch here but it fails to fully apply with the latest version. In addition, when you do get it installed you end up with errors like this:
[tom@tomcat ~]$ esxcfg-vmhbadevs
Can't locate VMware/VIRuntime.pm in @INC (@INC contains: /usr/lib64/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/local/lib64/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/site_perl/5.10.0 /usr/lib64/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl /usr/local/lib/perl5/site_perl/5.10.0 /usr/local/lib/perl5/site_perl /usr/lib/perl5/site_perl .) at /usr/local/bin/esxcfg-vmhbadevs line 10.
BEGIN failed--compilation aborted at /usr/local/bin/esxcfg-vmhbadevs line 10.
I’ve updated the patch to fix these problems for anyone interested.
vmware-rcli-distrib-x86_64.diff
I recently decided to virtualize my computers here at home using VMware’s ESXi. When I moved I realized I had a little too much computer equipment. Not to mention when all of them were running my electric bill would get a little ridiculous while these machines mostly idled. I ended up purchasing an HP Proliant ML115 G5 which is HP’s entry level tower server. They’re not officially supported by ESXi but after some googling I found the ML110′s (Intel) and ML115′s (AMD) have become popular vmware hosts for small setups. The only problem I had which I knew before purchasing it was the Nvidia MCP55 SATA controller was a pain. Most of the instructions I found for working around it were for ESX and not relevant to ESXi. While ESX does not recognize the MCP55 and needs PCI ID and kernel module information, ESXi does recognize the controller but will not let you use it. The problem is ESXi only recognizes PATA interfaces and loads the ide module instead of sata_nv and the installer blocks installs on IDE connected drives. I found the following patch here in the comments section which works around this.
ESXi_IDE-TargetFilter.diff
Just remember that you need to patch the file within install.tgz and can’t use oem.tgz on the installer image for this.