Difficult experience trying to create decent Web environment

Goals:

1. Put service configurations (Apache, JRun, etc.) from production Web servers into Subversion repositories.
2. Duplicate the Subversion repositories at a remote location for data security purposes.

These are pretty modest goals, as I'm sure anyone would agree. However, there is a litany of obstacles.

Circumstances:

1. I don't have shell access to the Web server OS.
2. The OS is Windows 2000.
3. I can RDC into the boxes but the only thing I can do is shut down or restart the machine.
4. I have SFTP access.
5. The SFTP server is VShell and appears to behave significantly differently than OpenSSH.
6. The "server administrators" are unwilling to install a Subversion client on the remote machines. (Of course, since I can't access the OS anyway then I couldn't operate a SVN client even if they were to deign to install one.)
   

I decided that a solution would be to mirror the remote filesystem on my local machine, in a working copy, and then commit the changes to a Subversion repository on my machine. I would then regularly dump the repository to a remote server via SSH.

That turned out to be tricky. I used lftp (which is awesome) for mirroring the remote filesystems to my local machine but I felt like the filtering/matching mechanisms were not flexible enough for my purposes. I ended up using lftp because using rsync over SSH to a Windows OS running Van Dyke VShell SFTP server does not work. The reason it does not work is because the "server engineers" disabled SSH on VShell. lftp operates over SFTP, as well as a zillion other protocols, so it worked perfectly.

That happened over the course of several months and was reasonably complete a few months ago. A couple weeks ago, I ended up using fuse and sshfs to mount all of the remote filesystems locally so I could use bash and GNU utilities on them. It dawned on me that this might allow me to use my local svn on the locally mounted remote Web servers!!! None of that mirroring crap!

Oh, were it so simple. I realized that there is a problem with renaming file with svn over sshfs and I just could not get it to work properly, even with the "-o workaround=all" option to sshfs. I thought, "Okay, whatever, I can try git." I recently started using git for some local development and thought that perhaps the way it uses the filesystem would not be impacted by the POSIX versus SFTP renaming problem. WELL . . . it was impacted alright. Of course. That's when I realized that since the filesystems are locally mounted that might allow me to try using rsync again!

rsync worked almost perfectly right away. The only thing was I discovered some corruption in the form of incomplete files. For example, I opened a file over sshfs with vi and the end of the file was truncated and filled with garbage characters. I opened the same file with jEdit and saw the same thing. I opened the same file with jEdit *over sftp rather than sshfs* and there was no corruption. I tried different sshfs and fuse options but it was when I disabled caching, "-o cache=no" that the corruption disappeared.

I just used some rsync filters to protect the .svn directories from deletion and the mirroring went fine. Previously, with lftp, I had to do some pretty convoluted things in bash to ensure that the local .svn directories were not deleted during mirroring.

So, it finally works, despite the best efforts of the "server engineers" and "server administrators" and "IT professionals".

I'm a Red Hat Certified Engineer and as of July 12, 2008 I'm still looking for a professional environment in which to work. Feel free to offer me a position with your organization that embraces best practices! :-)