Thu, 27 Feb 2003
I've been wrestling with a problem at work along with our vendor. We have a product on Win2K Advanced Server that kept blowing up during the install very earlier in the process. Some DLL call in Installshield was mailformatted in some fashion but we couldn't find why or what. We kept re-installing and removing hardware and diddling with registry entries and DCOM stuff to no avail.
Today however, I found on Installshield's site a direct reference to our error. We finally have a root cause and probable fix. The fix, though, requires the vendor to write some code. But at least its a fix. I'm definitely anxious to get moving forward again on this.
And of course I will get high visibility for being the miracle worker which feels pretty good. I love being the guy who does the magic voodoo. I suppose that's why I do what I do for a living. Between that, getting squirrelmail installed for my employer, tweaking this site, and getting the new nolug site set up, I've had a pretty darn good career day.
Installed Squirrelmail today for the company. I first used ApacheToolBox to build and install Apache with php4, SSL support, and even IMAP. I've used that a few times and this is the first time I had a build go with so little trouble. The only thing I had to do was install GNU sed to fix a long path issue during the PHP4 build. After that everything was automatic.
As a bonus, I found that Squirrelmail had a plugin to change user passwords through PAM. Very cool! Once we get LDAP rolling we can still use this to change passwords and we'll have a company-wide contact list in the webmail interface. Frankly, I'm surprised how smooth this went.
Actually the change password plugin had one wrinkle for Solaris. In the source for poppassd.c I had to change an include from "security/pam_misc.h" to "security/pam_module.h". Once I did that it built fine and ran after a little naggling with tcpd.
Well I rolled out NOLUG for beta test today (temporarily at http://www.nolug.org/postnuke/html. All comments have been complementary so far. It'd be nice to see NOLUG get moving again. I even impressed myself with the site actually. I managed to get a static content plugin installed for meeting documents archive as well as a calendar plugin and a separate headline grabber/RSS aggregator for them to play with.
I'll probably duplicate some of my content from here onto NOLUG as well where it's relevant. Now if I can just get em to move the meeting night I can start going again....
In addition to that I played with the blosxom.cgi a bit. I installed 2.0beta so I could play with plugins. Hence the Breadcrumbs at the top of the screen and the Calendar with archive links. I also played with Blagg a bit though I don't really have anything I'd want to syndicate into my blog though I may work on getting it to syndicate a bunch of stuff daily to email to me (security notices, etc) and perhaps integrate in to the sidebar as headlines rather than in the main blog.
I use Blosxom to run this here blog. It's simple to set up and does everything I need. And it also means I can write entries in a text editor "like God intended". I looked at some of the various blog services but those didn't interest me. I have an apache server and a domain after all. And of course I preferred something Open Source. Blosxom fits the bill
One way to post is of course using good ole vim/vi inside an ssh session.
But instead I work on my laptop remotely where I have a mirror of my blog
directory structure. I use rsync to sync
up the whole shebang. It looks like this from inside my local mirror of the blog:
rsync --rsh="ssh" -avz --progress . www.scottharney.com:/path/to/blog/scotth
Pics from this year's Barkus parade are up. Barkus is a parade just for dogs benefitting the Louisiana SPCA. There are 1500 dogs in the parade. It is enormous and great fun. We had beautiful weather and enjoyed dressing up as Elvis and his Memphis Mafia to complent the parade's "TailHouse Rock" theme.
Well, if for no other reason, everyone else is blogging (I really hate that term, actually). Actually I really needed a home for some documentation notes for myself. I've always kept little README files around noting changes I've made and things I discovered but it's really better to have them in one place like this.
Of course it's also for some amusement. And I have this journalism degree that I never actually use. Perhaps this will be useful to others as well. I also will likely cross-post some bits on nolug.org which is a site I manage as well.
I was also making some notes ranting about issues at work. On reflection, I decided to remove those for several reasons. Even though I elided all identifying details, I do work at a government facility and I'm under NDA. Better safe than sorry. And I also felt maybe I was giving a false view of my work environment. It's not all 'Dilbert-esque' but that's likely all that would be posted here.Fri, 21 Feb 2003
"I find your lack of clue disturbing"(Think Darth Vader in the first Star Wars)
I've recently done several remote upgrades of FreeBSD and wanted to note the procedure since it differs a bit from the handbook procedure for doing so. Still, you must review and understand the details of the procedure here
I use cvsup to get my copy of the stable source. Used /etc/standard-supfile provided with
cvsup to get it going to to update sources to 4.7-STABLE.
Here's the procedure:
The latter steps are what differ from the handbook procedures. You've got to really be careful doing this sort of thing remotely. Especially depending on how physically remote you are and especially with a firewall!
After completing the upgrade, I applied the necessary errata to correct any post 4.7-STABLE security and stability fixes.
Recently at work I needed a way to quickly recover systems from bare metal to a known-good state. These were, of course, Windows2000 boxes. Now there are several commercial solutions to "ghost" a system, but I didn't have funds available to do that.
Enter knoppix. Knoppix is a bootable CDROM distribution of Linux. It has many many uses and is worth investigating. A copy of it stays in my permanent SysAdmin toolchest. One thing it is particularly good at is autodetecting even the most recent hardware. I was pleasantly surprised to find it booted successfully with no manual intervention from me on our Dell PowerEdge 6650's with newish RAID and Gigabit Ethernet devices on board.
Once I booted up, rather than using the traditional unix dd(1) approach to make bit-for-bit drive images, I used the included partimage utility to clone those NTFS partitions. Since partimage knows about filesystem types, it's much more efficient than a bit-for-bit copy. I also had a complication in that I was saving to an temporarily NFS-mounted directory on a Sun server. The details of my procedure, including Linux-Solaris NFS tricks are below
Backing Up Images
Of course you need to get your network connections sorted out first. Boot the server with the Knoppix CD-ROM. Knoppix will come up in a full graphical environment. You, however, will need to work in a terminal console. On the panel bar at the bottom of the screen there is an app called "konsole" (just hold the mouse of the icons to identify them). One click will start it.
Use the following commands to set up network connectivity: First get root
$ sudo bashIf you don't have a dhcp server on your network statically configure your network card with appropriate parameters
# killall pump # ifconfig eth1 192.168.1.254 # ping 192.168.1.13 PING 192.168.1.13: 56 data bytes 64 bytes from nfs-server (192.168.1.13): icmp_seq=0. time=0. ms 64 bytes from nfs-server (192.168.1.13): icmp_seq=1. time=0. ms(use Ctrl-C to stop)
start portmap to enable nfs client connectivity
# portmapmount the drive. Yes this is a hairy command line but allows for reasonable performance with a Linux client connecting to a Solaris server
# mount -t nfs -o nfsvers=2,rsize=8192,wsize=8192,hard,intr \ 192.168.1.13:/win2k-image-dir /mnt(this mount command is all on one line)
get a snapsnot of disk partition layout.
# sfdisk -d > /mnt/server-partlayout.sfCopy the mbr of the array, just for good measure.
# dd if=/dev/sda of=/mnt/server-bootblock.img count=1 bs=512now use partimage to create your backups (we'll get to restore in a moment since it's largely these steps in reverse order). Partimage is a free app to image the disk partitions. Master documentation is at http://www.partimage.org
# partimagepartimage presents a GUI of sorts upon starting up. Use
Select the first partition in the list, then tab or arrow to the filename entry blank and fill in the path to the backup you are going to create like this "/mnt/server-hda1.partimage.gz". Since create backup is already selected, press F5 to get to the next dialog.
All the options in the next dialog are correct as well by default; ie. We want gzip compression for our image. Press F5 to continue. The next dialog lets you make an optional description. The next press of F5 will pause for a bit and then present you with a little information about the image it is about to create. Press enter and it begins. The length of time this takes depends on how full your partition is.
On NTFS partitions you will get the warning "NTFS support is experimental!". You can press enter and ignore this. If it makes the image successfully, it will restore successfully. I have tested this fully.
Work through the remainder of your drive(s) partitions this way.
That's it. Press the 'K' button and 'logout' to logout of knoppix. It will commence a shutdown and eject the disk. Do so, press enter, and Ctrl-Alt-Delete to reboot into Win2k
First boot the system. In my case with the Dells I had to set up the hardware RAID array on my PERC controller. I was, of course, smart enough to document how my RAID was laid out in advance.
Now boot the CD. Start by restoring the disk partition information
# sfdisk < /mnt/server-partlayout.sf # sfdisk -l(this will list the partition info in human readable format, just to verify)
Restore the bootblock image. This is probably not necessary, actually, but do it anyway.
# dd if=/mnt/server-bootblock.img \ of=/dev/sda count=1 bs=512(this all one line)
fire up partimage
It's the same basic steps as before. Except this time you're restoring So just change the appropriate checkbox to restore instead of backup. Make sure you select your partitions and their matching image filenames correctly (You documented this in writing when you backed them up didn't you?!). The on-screen prompts should be obvious. Restoring seems to go quicker than backing up.
There's one final step. Fire up partimage and select the partition that's marked as "bootable" and it's matching image filename. This time, choose "restore MBR information from image file" before pressing F5. Press F5 again to accept the defaults on the next screen. This restores your MBR on the bootable (aka "active") partition. On Win2K, that's your C: drive
Log out of Knoppix to initiate a reboot.You'll boot into a restored system partition. Then use appropriate software to restore any deltas from tape (Cause you're doing backups of course)
A Note on Setting up NFS
Setting up nfs differs immensely from Unix OS to Unix Os. So you need to get up to speed on that if you're saving your partitions to nfs like I did. On the Sun I had, I set the nfs /etc/dfs/dfstab config up properly as defined in the handy comments. With that done, as root I just fire up the necessary services (I leave this stuff off by default for security reasons)
# /etc/init.d/rpc start # /etc/init.d/nfs.server start # share - /share rw=192.168.1.254 "my share comment"(just verifies the share is available)
When you're done backing up or restoring.
# /etc/init.d/rpc stop # /etc/init.d/nfs.server stop # share(no output cause nothing is shared)
Make sure that you restore with the same Knoppix CD you backed up with. I discovered that partimage versions changed and later versions were not reading images created by the earlier versions.
As I said above, make sure you're documenting and testing your procedures. This is what works for me in one particular environment. It's not a cookie-cutter solution so you'll need to adapt to fit. I'm just glad to have a free, efficient way of backing up those annoying winboxen for the trouble that seems inevitable with that OS.
So you have just one IP address and a bunch of machines behind NAT. You've got port redirection working so your interal webserver behind the firewall is serving pages. But now you've got a second box that you need to host content on. Perhaps you have need to have a separate webserver running mod_perl and one running php. Or perhaps you've got (God forbid) an IIS box. And you don't want to redirect alternate ports. Here's away to have multiple webservers behind a single external IP address all running on Port 80.
What you need is a reverse inbound proxy established on your firewall. Apache with mod_proxy built and enabled does the trick.
First and foremost you need to have your DNS sorted out. I have both external and internal DNS servers. Bind 9 can do this with "views" though I personally have a preference for setting up djbdns. If you do not have an internal DNS for your domains then you'll need to reference your internal boxes by IP address in your apache configuration (see below).
The next thing to do is fix your firewall. You need to install apache. mod_proxy should come with it. You need to stop redirected port 80 inbound in your NAT (aka IP Masq) configuration since the firewall will now answer on Port 80. Since I have internal DNS servers, I also made sure my firewall's /etc/resolv.conf pointed to the internal DNS server.
Now you set up Apache on your firewall. Just do a basic configuration. Here's the magical lines snipped from httpd.conf
LoadModule proxy_module libexec/apache/libproxy.so AddModule mod_proxy.c NameVirtualHost your.external.ip.address <VirtualHost your.external.ip.address> ServerAdmin email@example.com ServerName www.yourwebsite.net ProxyPass / http://www.yourwebsite.net/ ProxyPassReverse / http://www.yourwebsite.net/ ErrorLog /var/log/apache/yourwebsite.net/error_log TransferLog /var/log/apache/yourwebsite.net/access_log <VirtualHost>
Since the internal DNS server has a local (ie 192.168.x.x) address for "www.yourwebsite.net", requests to that NameVirtualHost go to the appropriate internal box. And it need not be running apache. Anything that speaks http will be transparently proxied.
If you don't do internal DNS you'd replace "http://www.yourwebsite.net" with something like "http://192.168.5.80" where that is the IP of the internal server that you want to answer for www.yourwebsite.net.
Note that you can also do SSL https connections this way. The key is that you need to have your SSL certs and keyfiles on the firewall. The firewall would then speak standard http on port 80 to the internal box The config looks like this:
<VirtualHost your.external.ip.address:443> ServerAdmin firstname.lastname@example.org ServerName secure.yourwebsite.net ProxyPass / http://secure.yourwebsite.net/ ProxyPassReverse / http://secure.yourwebsite.net/ SSLEngine on SSLCertificateFile /path/to/certfile SSLCertificateKeyfile /ditto/for/keyfile.key ErrorLog /var/log/apache/secure.yourwebsite.net/error_log TransferLog /var/log/apache/secure.yourwebsite.net/access_log </VirtualHost>
As you can see, it really helps to have internal DNS set up. That makes things easier and allows you to have NameVirtualHosts on your internal boxes. You could just to IP based VirtualHosts internally configuring multiple 192.168.x.x IPs on your internal servers.
I'm sure you can imagine some very useful ways of doing this. It makes a test and development environment easy. You can stand up a replacement website without going through the hastle of waiting for public DNS to "catch up".
Obviously there are security considerations. I won't go into a major discussion about that here except to say that you need to think about it. For my needs, this increased my security posture because I could move Win2000 machines with many potential vulnerabilities behind the firewall and reduce exposure to just IIS and cross-site scripting issues. That's still plenty to worry about, but better than having, say, MSSQL outside your firewall)
Another implication of this is that your logging of website connections changes. All that your internal boxes will ever log now are connections from the firewall. So those logs are useless for tracking site traffic, etc. But all your hits are logged -- separately the way I configured it -- on the firewall itself. Just make sure you make those log subdirectories manually before restarting apache because apache won't create them. The master apache error log will report this, of course.
To debug a script try running "sh -v scriptname"
Thanks to J. Kent Busbee on the nolug mailing list for this quickie.