Fri, 21 Feb 2003
"I find your lack of clue disturbing"(Think Darth Vader in the first Star Wars)
I've recently done several remote upgrades of FreeBSD and wanted to note the procedure since it differs a bit from the handbook procedure for doing so. Still, you must review and understand the details of the procedure here
I use cvsup to get my copy of the stable source. Used /etc/standard-supfile provided with
cvsup to get it going to to update sources to 4.7-STABLE.
Here's the procedure:
The latter steps are what differ from the handbook procedures. You've got to really be careful doing this sort of thing remotely. Especially depending on how physically remote you are and especially with a firewall!
After completing the upgrade, I applied the necessary errata to correct any post 4.7-STABLE security and stability fixes.
Recently at work I needed a way to quickly recover systems from bare metal to a known-good state. These were, of course, Windows2000 boxes. Now there are several commercial solutions to "ghost" a system, but I didn't have funds available to do that.
Enter knoppix. Knoppix is a bootable CDROM distribution of Linux. It has many many uses and is worth investigating. A copy of it stays in my permanent SysAdmin toolchest. One thing it is particularly good at is autodetecting even the most recent hardware. I was pleasantly surprised to find it booted successfully with no manual intervention from me on our Dell PowerEdge 6650's with newish RAID and Gigabit Ethernet devices on board.
Once I booted up, rather than using the traditional unix dd(1) approach to make bit-for-bit drive images, I used the included partimage utility to clone those NTFS partitions. Since partimage knows about filesystem types, it's much more efficient than a bit-for-bit copy. I also had a complication in that I was saving to an temporarily NFS-mounted directory on a Sun server. The details of my procedure, including Linux-Solaris NFS tricks are below
Backing Up Images
Of course you need to get your network connections sorted out first. Boot the server with the Knoppix CD-ROM. Knoppix will come up in a full graphical environment. You, however, will need to work in a terminal console. On the panel bar at the bottom of the screen there is an app called "konsole" (just hold the mouse of the icons to identify them). One click will start it.
Use the following commands to set up network connectivity: First get root
$ sudo bashIf you don't have a dhcp server on your network statically configure your network card with appropriate parameters
# killall pump # ifconfig eth1 192.168.1.254 # ping 192.168.1.13 PING 192.168.1.13: 56 data bytes 64 bytes from nfs-server (192.168.1.13): icmp_seq=0. time=0. ms 64 bytes from nfs-server (192.168.1.13): icmp_seq=1. time=0. ms(use Ctrl-C to stop)
start portmap to enable nfs client connectivity
# portmapmount the drive. Yes this is a hairy command line but allows for reasonable performance with a Linux client connecting to a Solaris server
# mount -t nfs -o nfsvers=2,rsize=8192,wsize=8192,hard,intr \ 192.168.1.13:/win2k-image-dir /mnt(this mount command is all on one line)
get a snapsnot of disk partition layout.
# sfdisk -d > /mnt/server-partlayout.sfCopy the mbr of the array, just for good measure.
# dd if=/dev/sda of=/mnt/server-bootblock.img count=1 bs=512now use partimage to create your backups (we'll get to restore in a moment since it's largely these steps in reverse order). Partimage is a free app to image the disk partitions. Master documentation is at http://www.partimage.org
# partimagepartimage presents a GUI of sorts upon starting up. Use
Select the first partition in the list, then tab or arrow to the filename entry blank and fill in the path to the backup you are going to create like this "/mnt/server-hda1.partimage.gz". Since create backup is already selected, press F5 to get to the next dialog.
All the options in the next dialog are correct as well by default; ie. We want gzip compression for our image. Press F5 to continue. The next dialog lets you make an optional description. The next press of F5 will pause for a bit and then present you with a little information about the image it is about to create. Press enter and it begins. The length of time this takes depends on how full your partition is.
On NTFS partitions you will get the warning "NTFS support is experimental!". You can press enter and ignore this. If it makes the image successfully, it will restore successfully. I have tested this fully.
Work through the remainder of your drive(s) partitions this way.
That's it. Press the 'K' button and 'logout' to logout of knoppix. It will commence a shutdown and eject the disk. Do so, press enter, and Ctrl-Alt-Delete to reboot into Win2k
First boot the system. In my case with the Dells I had to set up the hardware RAID array on my PERC controller. I was, of course, smart enough to document how my RAID was laid out in advance.
Now boot the CD. Start by restoring the disk partition information
# sfdisk < /mnt/server-partlayout.sf # sfdisk -l(this will list the partition info in human readable format, just to verify)
Restore the bootblock image. This is probably not necessary, actually, but do it anyway.
# dd if=/mnt/server-bootblock.img \ of=/dev/sda count=1 bs=512(this all one line)
fire up partimage
It's the same basic steps as before. Except this time you're restoring So just change the appropriate checkbox to restore instead of backup. Make sure you select your partitions and their matching image filenames correctly (You documented this in writing when you backed them up didn't you?!). The on-screen prompts should be obvious. Restoring seems to go quicker than backing up.
There's one final step. Fire up partimage and select the partition that's marked as "bootable" and it's matching image filename. This time, choose "restore MBR information from image file" before pressing F5. Press F5 again to accept the defaults on the next screen. This restores your MBR on the bootable (aka "active") partition. On Win2K, that's your C: drive
Log out of Knoppix to initiate a reboot.You'll boot into a restored system partition. Then use appropriate software to restore any deltas from tape (Cause you're doing backups of course)
A Note on Setting up NFS
Setting up nfs differs immensely from Unix OS to Unix Os. So you need to get up to speed on that if you're saving your partitions to nfs like I did. On the Sun I had, I set the nfs /etc/dfs/dfstab config up properly as defined in the handy comments. With that done, as root I just fire up the necessary services (I leave this stuff off by default for security reasons)
# /etc/init.d/rpc start # /etc/init.d/nfs.server start # share - /share rw=192.168.1.254 "my share comment"(just verifies the share is available)
When you're done backing up or restoring.
# /etc/init.d/rpc stop # /etc/init.d/nfs.server stop # share(no output cause nothing is shared)
Make sure that you restore with the same Knoppix CD you backed up with. I discovered that partimage versions changed and later versions were not reading images created by the earlier versions.
As I said above, make sure you're documenting and testing your procedures. This is what works for me in one particular environment. It's not a cookie-cutter solution so you'll need to adapt to fit. I'm just glad to have a free, efficient way of backing up those annoying winboxen for the trouble that seems inevitable with that OS.
So you have just one IP address and a bunch of machines behind NAT. You've got port redirection working so your interal webserver behind the firewall is serving pages. But now you've got a second box that you need to host content on. Perhaps you have need to have a separate webserver running mod_perl and one running php. Or perhaps you've got (God forbid) an IIS box. And you don't want to redirect alternate ports. Here's away to have multiple webservers behind a single external IP address all running on Port 80.
What you need is a reverse inbound proxy established on your firewall. Apache with mod_proxy built and enabled does the trick.
First and foremost you need to have your DNS sorted out. I have both external and internal DNS servers. Bind 9 can do this with "views" though I personally have a preference for setting up djbdns. If you do not have an internal DNS for your domains then you'll need to reference your internal boxes by IP address in your apache configuration (see below).
The next thing to do is fix your firewall. You need to install apache. mod_proxy should come with it. You need to stop redirected port 80 inbound in your NAT (aka IP Masq) configuration since the firewall will now answer on Port 80. Since I have internal DNS servers, I also made sure my firewall's /etc/resolv.conf pointed to the internal DNS server.
Now you set up Apache on your firewall. Just do a basic configuration. Here's the magical lines snipped from httpd.conf
LoadModule proxy_module libexec/apache/libproxy.so AddModule mod_proxy.c NameVirtualHost your.external.ip.address <VirtualHost your.external.ip.address> ServerAdmin email@example.com ServerName www.yourwebsite.net ProxyPass / http://www.yourwebsite.net/ ProxyPassReverse / http://www.yourwebsite.net/ ErrorLog /var/log/apache/yourwebsite.net/error_log TransferLog /var/log/apache/yourwebsite.net/access_log <VirtualHost>
Since the internal DNS server has a local (ie 192.168.x.x) address for "www.yourwebsite.net", requests to that NameVirtualHost go to the appropriate internal box. And it need not be running apache. Anything that speaks http will be transparently proxied.
If you don't do internal DNS you'd replace "http://www.yourwebsite.net" with something like "http://192.168.5.80" where that is the IP of the internal server that you want to answer for www.yourwebsite.net.
Note that you can also do SSL https connections this way. The key is that you need to have your SSL certs and keyfiles on the firewall. The firewall would then speak standard http on port 80 to the internal box The config looks like this:
<VirtualHost your.external.ip.address:443> ServerAdmin firstname.lastname@example.org ServerName secure.yourwebsite.net ProxyPass / http://secure.yourwebsite.net/ ProxyPassReverse / http://secure.yourwebsite.net/ SSLEngine on SSLCertificateFile /path/to/certfile SSLCertificateKeyfile /ditto/for/keyfile.key ErrorLog /var/log/apache/secure.yourwebsite.net/error_log TransferLog /var/log/apache/secure.yourwebsite.net/access_log </VirtualHost>
As you can see, it really helps to have internal DNS set up. That makes things easier and allows you to have NameVirtualHosts on your internal boxes. You could just to IP based VirtualHosts internally configuring multiple 192.168.x.x IPs on your internal servers.
I'm sure you can imagine some very useful ways of doing this. It makes a test and development environment easy. You can stand up a replacement website without going through the hastle of waiting for public DNS to "catch up".
Obviously there are security considerations. I won't go into a major discussion about that here except to say that you need to think about it. For my needs, this increased my security posture because I could move Win2000 machines with many potential vulnerabilities behind the firewall and reduce exposure to just IIS and cross-site scripting issues. That's still plenty to worry about, but better than having, say, MSSQL outside your firewall)
Another implication of this is that your logging of website connections changes. All that your internal boxes will ever log now are connections from the firewall. So those logs are useless for tracking site traffic, etc. But all your hits are logged -- separately the way I configured it -- on the firewall itself. Just make sure you make those log subdirectories manually before restarting apache because apache won't create them. The master apache error log will report this, of course.
To debug a script try running "sh -v scriptname"
Thanks to J. Kent Busbee on the nolug mailing list for this quickie.