Monday, January 23, 2017

Need to import things from JIRA into Google sheets?

Attach this to a google sheet and then just pass in the filter number.

This should get past both jira's 1000 issue limit and importxml's limit. The only limitation is script runtime which is 5 minutes. Have imported up to 1500 issues with no problem.

function parseXml(searchNumber) {
  var searchURL = 'https://jira.wherever.com/sr/jira.issueviews:searchrequest-xml/' + searchNumber + '/SearchRequest-' + searchNumber + '.xml?tempMax=500&os_username=username&os_password=password';
  var xml = UrlFetchApp.fetch(searchURL).getContentText();
  var document = XmlService.parse(xml);
  
  // get total
  var total = document.getContent(2).getChild("channel").getChild("issue").getAttribute("total").getValue();
  var pages = Math.ceil(total/500);
  
  var sheetData = new Array();
  
  //loop through pages
  for (var i = 0; i < pages; i++)
  {
    var pageURL = searchURL + '&pager/start=' + i * 500;
    var xml = UrlFetchApp.fetch(pageURL).getContentText();
    var document = XmlService.parse(xml);
    var items = document.getContent(2).getChild("channel").getChildren("item");
    
    var xmlElementNames = [
                             "title", 
                             "link",
                             "project",
                             "description",
                             "key",
                             "summary",
                             "type",
                             "priority",
                             "status",
                             "resolution",
                             "assignee",
                             "reporter",
                             "created",
                             "updated",
                             "fixVersion",
                             "timeoriginalestimate",
                             "timeestimate",
                             "comments",
                             "Sprint"
                            ]
    
    sheetData.push(xmlElementNames);
    
    for (var j = 0; j < items.length; j++) 
    {
      var rowData = [];
      for (var k = 0; k < xmlElementNames.length; k++)
      {
        var itemElement = items[j].getChild(xmlElementNames[k]);
        if( itemElement )
        {
          rowData.push(itemElement.getText());
        } 
        else
        {
          // check the customfields
          var customfields = items[j].getChild("customfields").getChildren("customfield");
          var customfieldfound = false;
          for (var l = 0; l < customfields.length; l++)
          {
            if( customfields[l].getChild("customfieldname").getValue() == xmlElementNames[k] )
            {
              rowData.push(customfields[l].getChild("customfieldvalues").getChild("customfieldvalue").getText());
              customfieldfound = true;
            }
          }
          if( customfieldfound == false )
          {
            rowData.push("");
          }
        }
      }
      sheetData.push( rowData );
      //Logger.log(items[j].getChild("title").getText());
    }
  }
  
  var sheet = SpreadsheetApp.getActiveSheet();
  sheet.clear();
  var sheetRange = sheet.getRange(1, 1, sheetData.length, sheetData[0].length );
  sheetRange.setValues(sheetData);
  
}

Wednesday, September 28, 2016

Need a dump?

If you ever need a crash dump from a remote machine because you're trying to debug a crash.

Create a key at 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps
No settings required. Just the key's existence is enough 

Default settings will put the dump in to %LOCALAPPDATA%\CrashDumps

For more info : Here.

Tuesday, December 2, 2014

WASP x64 ( Windows x64 , Apache 2.4 x64 , SVN, PHP )

To make things easy WASP will now go out in the wild.

Ever wanted a 64 bit distribution of apache 2.4, svn and php but putting all the parts together caused butt pain?
Well here you go.

How stable is it?
It hasn't crashed in production. YET.

How is it configured?
httpd.conf. It uses lots of memory! It eats 2.3 GB of RAM, and then holds steady due to a 2GB svn cache. If you have a small repo and more RAM? MAKE the cache BIGGER!

In production it currently serves a 1.6TB svn repo with 465k revisions that is growing, and acts as a proxy for multiple internal web servers. It also runs PHP so you can use it to do other useful things. Uses rotatelogs to make daily rotating logs, cause usually no one sets this up either!

Why did I put it together?
Because nothing out there fit my needs.

Why did I put this here?
Because hopefully it'll benefit someone else that also wants to roll their own and not use collabnet/wandisco/visualsvn. They are all great products, but not quite what I'm looking for.

How do I install it?
Extract to C:\WASP. Run httpd.exe

Download




Monday, July 21, 2014

Note to self : 0x000001d error

Usually means some /arch: for code generation that is not supported, like SSE2 / AVX

Monday, June 9, 2014

Beware the Windows Scheduled Tasks!

I've had a TeamCity server run much much slower than it's other counterparts.

I finally narrowed it down to Scheduled Tasks. The agent was started at logon by Scheduled Tasks.

It looks like Scheduled Tasks will start in under below normal priority and while this wasn't a problem for many of our build tools, TortoiseSVN however performs very very poorly in below normal priority.

Simple Hall and Teamcity Integration

You will have to set up these 2 parameters as environment variables to be passed in your TeamCity project parameters configuration.

env.BuildID = %teamcity.build.id%
env.BuildTypeID = %system.teamcity.buildType.id%

Checks to see if it's running under TeamCity and then sends the INFO

if not "%TEAMCITY_PROJECT_NAME%" == "" curl -X POST -H "Content-Type: application/json" -d "{\"title\":\"TeamCity\",\"message\":\"Failed - %TEAMCITY_PROJECT_NAME% - %TEAMCITY_BUILDCONF_NAME% - %BUILD_NUMBER% ^<a href="http://TEAMCITYURL/viewLog.html?buildId=%BuildID%&buildTypeId=%BuildTypeID%&tab=buildResultsDiv"^>View Build^</a^>\",\"picture\":\"http:/TEAMCITYURL/img/icons/TeamCity32.png\"}" https://hall.com/api/1/services/generic/HALLID

Thursday, June 5, 2014

Visual Studio Property files and line endings

If you like to lay out multi line preprocessor definition in macros in .props files.

<pc32_preprocessor>
      WF_PC32;
      WIN32;
      _CRT_SECURE_NO_WARNINGS;
      $(FORCED_ENVARS)
</pc32_preprocessor>

Something like that. Certain plugins like the clang plugin will cause all sorts of interesting errors because it'll pass the command line to the compiler with the CRLF.

Change line endings to Old Mac format CR only.

VS will allow you to edit and it will respect the line endings and wont convert it, and it seems to pass through to the compiler just fine.

Integrating windows hosted SVN with hall, logs and diffs provided with WEBSVN

For posterity

SET PATH=%PATH%;E:\svn;E:\curl\bin;E:\bin

SET REPOS="%1"
SET REV=%2

SET HALLROOM=put Group API Token here
SET WEBSVNHOST=https://YOUR_URL_HERE/revision.php?repname=REPONAME
SET PATHTOMATCH=trunk

REM HALL INTEGRATION

svnlook author %REPOS% --revision %REV%> E:\temp\svnlook%REV%_USER
SET /P SVN_USERNAME=<E:\temp\svnlook%REV%_USER
del E:\temp\svnlook%REV%_USER

svnlook log %REPOS% --revision %REV%> E:\temp\svnlook%REV%_LOG
SET /P SVN_COMMIT_MESSAGE=<E:\temp\svnlook%REV%_LOG
del E:\temp\svnlook%REV%_LOG

svnlook dirs-changed %REPOS% --revision %REV%> E:\temp\svnlook%REV%_CHANGED
SET /P SVN_COMMIT_DIRECTORY=<E:\temp\svnlook%REV%_CHANGED

SET SVNMESSAGE=Revision : %REV% - %SVN_USERNAME% - %SVN_COMMIT_MESSAGE%

grep -E "%PATHTOMATCH%" E:\temp\svnlook%REV%_CHANGED
if "%ERRORLEVEL%"=="0" curl -X POST -H "Content-Type: application/json" -d "{\"title\":\"SVN Commit\",\"message\":\"%SVNMESSAGE%^<br^> ^<a href="%WEBSVNHOST%&path=%%2F&rev=%REV%"^>Inspect Log^</a^>\"}" https://hall.com/api/1/services/generic/%HALLROOM%

del E:\temp\svnlook%REV%_CHANGED

Wednesday, May 1, 2013

Reset WINS cache on a QNAP device

Disable WINS server from web administration page

SSH into it

cd /var/log/samba/locks

delete wins.dat

Enable WINS server from web administration page

Done.

Wednesday, October 17, 2012

VSPTree is alive!

I've been using Windows 8 for the last couple of months and have been having a great time with it. It's Windows 7 with a nice flat UI, much faster I/O. Stability is the same, and pretty much have never opened a metro app. I'm not even sure how to use those correctly. The new Start Menu is just a very big start menu to me.

Anyway I digress, I needed to profile something and went with the usual suspects. CodeAnalyst, Got no data. AQTime, failed to launch the exe. Very Sleepy, worked but no hierarchical call graph. Tried to resurrect VSPTree with the VS2010 profiler and I got a driver not allowed error.

Oh well. Some digging around and I found Remote Tools for Visual Studio 2012.

So I took a little time and updated VSPTree to work with it, and added some minor usability updates. It's not the best application in the world, but sometimes it's handy just to have it.

VSPTree 0.5 : Now with Windows 8 64 bit support. Here.

Friday, October 12, 2012

Conditionals in Property Sheets

I've since moved to using property sheets exclusively instead of wfMake and it's been going well. A bit more more typing but I like the fact that I can edit a bunch of includes and not have to regenerate the vcproj/vcxproj. The most time consuming part was going through all the projects and setting the right macros everywhere. But you only have to do it once.

It's one place to update everything and can be syntax highlighted and formatted quite neatly.

You end up with something like this. This is only an excerpt, the whole thing is quite huge. You can then string your macros any way you like. The fanciest bit of this is the conditionals at the top that I use because some sln's that include these projects are sometimes in different places, and there was no other way I could think of to correctly set the path.



  
  
    
      
        $(ProjectDir)..\..\..\
      
    
    
      
        $(ProjectDir)..\..\..\
      
    
    
      
        $(SolutionDir)
      
    
  
  
    
    
      WF_DEBUG;
      _DEBUG
    

    
      WF_RELEASE;
      NDEBUG
    

    
      WF_FINAL;
      NDEBUG
    

    
    
      WF_PC32;
      WIN32;
      _CRT_SECURE_NO_WARNINGS;
      $(FORCED_ENVARS)
    

    
      $(WF_DEBUG);
      $(PC32_PREPROCESSOR);
      __PLACEMENT_NEW_INLINE
    


  


Wednesday, March 28, 2012

Pre-commit goodness

This solution was used to solve programmers frequently accidentally committing debug executables into SVN. On developer machines you rarely see this issue because all the debug libs exist, but a level designer or artist will frequently run into the problem and will have no idea what the problem is, and someone will have to take a look at it and figure out what in the world is going on. Then the doh! moment happens and someone has to recompile and commit that.

The precommit hook also show an example of how to deny committing certain files like .argb and thumb.db

The solution is simple enough. Add a precommit hook that scans all the exe's for debug libs. This method uses Dependency Walker by Steve Miller.

So far I've tested it on rather large commits with 20 exe's and it doesn't take more than 20 - 30 seconds to complete. Server is running a i7-960 for reference.

cd /d %~dp0
SETLOCAL ENABLEDELAYEDEXPANSION
SET REPOS=%1
SET TXN=%2

SET PATH=E:\svn;E:\bin;E:\depends

svnlook changed %REPOS% -t %TXN% > E:\temp\svnlook%TXN%

grep -E "A.+(.argb)" E:\temp\svnlook%TXN%

if "%ERRORLEVEL%"=="0" (
 echo Do not commit ARGB files! >&2
 goto :error
) 

grep -E "A.+(thumbs.db)" E:\temp\svnlook%TXN%

if "%ERRORLEVEL%"=="0" (
 echo Do not commit thumbs.db! >&2
 goto :error
)

grep -P -o "(?<=\w\s\s\s).+\.exe" E:\temp\svnlook%TXN% > E:\temp\svnlook%TXN%_exegrep

for /f "tokens=1 delims=¶" %%f IN (E:\temp\svnlook%TXN%_exegrep) DO (
 svnlook cat %REPOS% -t %TXN% "%%f" > E:\temp\temp
 depends.exe /c /of:E:\temp\dep.txt E:\temp\temp
 grep -E "QT.+D4.DLL" E:\temp\dep.txt
 if "!ERRORLEVEL!"=="0" (
  echo Do not commit Debug executables! %%f is in debug>&2
  goto :error
 )
)

del E:\temp\svnlook%TXN%
del E:\temp\svnlook%TXN%_exegrep
exit 0

:error
del E:\temp\svnlook%TXN%
del E:\temp\svnlook%TXN%_exegrep
exit 1

Wednesday, February 15, 2012

Saving tortoiseGit password

Took me a little while to find this, so I thought I would immortalize it here.

Apparently you need to save a file called _netrc in %USERPROFILE%

So a quick little batch file :

@echo off
SET /P HOST=What is the hostname i.e github.com?
SET /P USERNAME=What is your username?
SET /P PASSWORD=What is your password?

echo machine %HOST% login %USERNAME% password %PASSWORD% > "%USERPROFILE%\_netrc"

Thursday, February 9, 2012

Crappy documentation? Internet to the rescue

So through some amazing series of events, I broke the TeamCity installation.

Things weren't building. I thought it might be a bug, so maybe I'll try a newer version. The newer version needed an upgrade to the database so I said OK, but I made sure to back up first of course.

Lo and behold the new version did solve the old problem, but it presented a host of new ones! Basically made it unusable. Of course this was a beta build, but hey beta is the new GMC nowadays right? WRONG!

Oh well, time to uninstall and go back to the backup. Cue the 30 minutes I spend trying to figure out how in the world do I restore this backup!

The manual says :

Just do this!

maintainDB[cmd|sh] restore -F <TeamCity backup file name>

Error : config is not empty. Ok I'll delete that then.
run again
Error : system is not empty. Ok I'll empty that too!
run again.
Needs a target!

Wuh? That's the "Restoring Data from Backup to Another Database" instruction not the standard restore.

But no. It doesn't work. Ok, scrounge around a bit more....

Hallelujah.

Turns out you need to copy the database.hsqldb.properties.dist out of the config directory in your backup zip first. Give that as the -T database.hsqldb.properties.dist

And then everything gets restored. Then it gives you some instruction that says, you have to copy that file into system\buildserver.properties.
Did I? Nope. Did the restore work? Yes.

Bad documentation is bad.

Tuesday, November 8, 2011

S3 and the 5GB limit!

s3cmd as of this point still does not support mutli-part uploads to break the 5GB barrier.

But http://sprightlysoft.com/S3Upload/ does!

Now the problem with that is the Etag that used to be the md5 is NOT the md5 anymore once you start using multi-part uploads. It looks like it, but it's not because it has a dash in it.

So a slight modification of s3cmd

@@ -633,7 +633,10 @@
  output(u"   File size: %s" % info['headers']['content-length'])
  output(u"   Last mod:  %s" % info['headers']['last-modified'])
  output(u"   MIME type: %s" % info['headers']['content-type'])
- output(u"   MD5 sum:   %s" % info['headers']['etag'].strip('"'))
+ if info["headers"].has_key("x-amz-meta-multipart-etag"):
+ output(u"   MD5 sum:   %s" % info['headers']['x-amz-meta-multipart-etag'])
+ else:
+ output(u"   MD5 sum:   %s" % info['headers']['etag'].strip('"'))
  else:
  info = s3.bucket_info(uri)
  output(u"%s (bucket):" % uri.uri())

and bam you're back in business with s3Upload and s3cmd

Executable version with Standalone Python 2.5 and cx_freeze : Here


Tuesday, November 1, 2011

New google reader interface

Are you kidding me? Why all the wasted space? It's TERRIBLE...

This would be so much better.

Friday, October 28, 2011

Thread management

Interesting article on the Bulldozer architecture today : http://techreport.com/articles.x/21865

So I thought, why not try this on an i7 as well.

So I tested it with the Euler3D BENCHMARK like they do with 4 different configurations.



Pretty interesting results.
8 cores is fastest, but not much faster due to diminishing returns probably from contention or the algorithm.
0F is slowest definitely because of contention.
55 affinity is faster than the windows scheduler. I noticed that not any one core every hit 100% while the windows scheduler was scheduling stuff, so the time loss could be from threads switching cores.

Tuesday, October 25, 2011

Arrrgh.. I always make my life difficult

A long long time ago, before subversion edge and wandisco ubersvn, there was a time when we had to set up our own svn servers.

I started with the standard apache distro, added SSL, then SVN on top of that. It took me a few days the first try but it was good.

Then eventually I got tired of it, and moved to XAMPP + SVN. And this was good for a long time.

And then.. XAMPP started using VC9 as their compiler, while win32svn stayed with VC6. And things were not GOOD.

So now I have to use XAMPP, download WANDISCO's vanilla svn, rip out their .so's and dump them into XAMPP. Now all is well.

Also XP seems to barf on SSL connections with the win32svn distro. So use WANDISCO's or sliksvn for your binaries if you're still on XP and need SSL.

Friday, October 7, 2011

wtMake

This is an internal tool  for configuration management that I decided would be great to put out in the world. I like build tools. Make, SCons, CMake, Qmake, ANT. I love what they do for you and what great programmers can do with them. I also like Visual Studio. A lot.

Most people tend to pick the build tool they want to use, then generate a vcproj or sln from them. This is great and all when you wanna target your software to build on almost any system with GCC, cl, xcode. Whatever.

For small projects it works great. When a project is large however, programmers like to put stuff in folders and get all organized. Then your configuration changes, and you need to regenerate that vcproj. BAM! It's all flat again. Unless I'm totally wrong and Qmake or Cmake can actually generate vcproj's with filters, I haven't found that documentation.

Truth be told, I work with several proprietary VS plugins for platforms that like to store settings and configurations within the vcproj. What happens when other build tools generate vcprojs? They kinda go missing as well. I could of course go figure out what they're adding and add all these options to the vcproj generators in the bu.... NOPE.

I'm not gonna go and remake all the settings that the fantastic engineers at Microsoft and Sony have done to make it easy for programmers to actually add their platforms as targets in Visual Studio. It's good work. Let's keep it! Programmers are accustomed to being able to look at the vcproj properties in VS and double click toggle boxes and drop down lists. This is good. Let's keep most of that! I'm also not going to make my own build tool from scratch either. Not when I want to make it easy for myself to let's say use incredibuild or another distributed build plugin for VS.

So how do I configure include directories and library directories and additional libs and all these fancy things in the vcproj in an easy and quick manner?! Your probably think this is an easy task, but it's kinda tedious when you have 4 libraries, 3 platforms, with 3 configurations each. That's... a lot of places to add things to. You could use VS property sheets. But, I actually found them kind of a pain to use... cause you're still buried in a ton of different VS menus, then you have to add it to a billion project configurations. Thanks, but no thanks.

The vcproj format is pretty simple and below is a very simplified version of the format. Basically each configuration in configuration element with a name as an attribute. Then every tool is in an element with its options are stored as attribute. In the example below, the VCCLCompilerTool stores preprocessor definitions in the PreprocessorDefinitions attribute. What wtMake does is actually looks through every tool element that has the right configuration and replaces the current attribute with what you set.

<Configuration Name="Release|Win32" >
	<Tool PreprocessorDefinitions="DEFINITION" Name="VCCLCompilerTool"/>
</Configuration>

So I wrote a little tool, that can modify almost any element in a vcproj's tool attribute to the things I need it to be. The syntax is simple and fast.

//comment
:rule
	#vsattribute
		< text you want to go in there.
	!forceattribute ? NameAttribute
		< text you want to go in there.

So let's say we wanted to add freetype to the vcproj. This is the set of rules that I would need to cover everything for freetype and PC.

:pc
	#PreprocessorDefinitions
		< WF_PC32
		< WIN32
		< _CRT_SECURE_NO_WARNINGS
	#AdditionalDependencies
		< Ws2_32.lib xinput.lib libpad.lib
	!AdditionalOptions ? VCCLCompilerTool | VCCLX360CompilerTool
		< /MP
	!MinimalRebuild ? VCCLCompilerTool | VCCLX360CompilerTool
		< false

// Freetype
:freetypeinclude
	#AdditionalIncludeDirectories
		< $(ProjectDir)..\..\..\3rdparty\freetype\include
:freetype360
	#AdditionalLibraryDirectories
		< 3rdparty\freetype\objs\xbox360\vc2008
:freetypepc
	#AdditionalLibraryDirectories
		< 3rdparty\freetype\objs\win32\vc2008
:freetypeps3
	#AdditionalLibraryDirectories
		< 3rdparty\freetype\objs\ps3\vc2008
:freetypedebug
	#AdditionalDependencies
		< freetype239MT_D.lib
:freetyperelease
	#AdditionalDependencies
		< freetype239MT.lib
:freetypedebugps3
	#AdditionalDependencies
		< freetype239MT_D.a
:freetypereleaseps3
	#AdditionalDependencies
		< freetype239MT.a

So now that's fine and dandy. How do we use the rules? Well then, now we have a project template. The format is something like this.

$templatename
	#all
		< rule
	#Configuration|Platform
		< rule

#all is a special tag so that it's added to all configuration.

Here is an example of just the pc portion of a game project template.

$mygame
	#all
		< bulletincludes
		< raknetcompiler
		< wfEngineSource
	#Debug|Win32
		< debug
		< pc
		< directxCompiler
		< directxdebug
		< freetypepc
		< freetypedebug
		< physxpcCompiler
		< physxpc
		< bulletpclibs
		< force
	#Release|Win32
		< release
		< pc
		< directxCompiler
		< directxrelease
		< freetypepc
		< freetyperelease
		< physxpcCompiler
		< physxpc
		< bulletpclibs
		< force
	#Final|Win32
		< final
		< pc
		< directxCompiler
		< directxrelease
		< freetypepc
		< freetyperelease
		< physxpcCompiler
		< physxpc
		< bulletpclibs
		< force

So how do you actually use the tool? Simple.

wtmake example.vcproj -t templatename

if you'd like to use it with the example above :

wtmake mygame.vcproj -t mygame

GOTCHAS!

The tool will completely replace all text in that attribute! So if you have a preexisting project you'd like to modify with it, make sure you have a backup in case you don't get all the configurations the first time!

The tool is modifies only elements that exist, if it doesn't find one. It doesn't add it. I wrote it this way to simplify the actual tool rule syntax. So you don't have to worry about the 100's of other Tool elements.

If you need to force an attribute, like for AdditionalOptions to add the all important /MP switch to cl. You can do that by using a ! for a forceattribute . It will create it if the element has an attribute called "name" that matches. This is the method I used to add /MP to only the VCCLCompiler tool and not to VCCustomBuildTool,VCMIDLTool and all the other ones.

If you are interested, you can download this tool from here. There is an example PC project with a bunch of libraries already preconfigured in the file called wtMakeConfig.

If you want source, please feel free to ask me. I may release it eventually after I clean it up of some dependencies of other libs. Released under zlib.

A perpetual SVN backup method to the cloud

I came up with this method a long time ago to provide redundancy and archival to the svn servers that I operate. I don't think I've found a similar solution online so I thought I would share because it works pretty well for me.

Basic Redundancy

The first step of course is to make sure that you provide a good amount of local redundancy. This is fairly standard procedure. RAID the drives the svn repo is on. Now some people might stop there and say, a mirror is good enough, like one company I remember that lost all it's data because someone disgruntled just did a DROP table in mySQL. For those of us who are more paranoid... we do not stop here.

The next step in my redundancy strategy for svn is to have a mirror server that uses svnsync to maintain a 2nd copy of the repo. If the main server fails, a simple DNS switch and everybody will be back in business. The mirror server runs on a perpetual loop, of syncing waiting a minute, then syncing again. I know that this is probably a little wasteful because it pings the main server so often, but in practice I've found it's much more stable than using post-commit-hooks to send a message to the mirror server. Some online examples of using svnsync tell you to use a post-commit-hook to call svnsync on the main server. I've found that this presents with 2 problems.

First problem that I encountered with that is, svn seems to be designed to not send a commit done message to the client until the entire post commit hook is done. So on particularly large commits, waiting for a sync to the mirror is wasting time on the client side.

The other problem, is that syncing by sending data from one Apache server to another Apache server, is extremely slow. Which is why I always call the svnsync on the mirror server using a UNC file path and not over https.

Archiving to the Cloud

So now I have a redundant main server, and a redundant mirror, that has at most a delay of a minute before it updates itself to the latest revision. Perfect right? Not yet. Any disaster plan should always have an offsite backup solution. I could use an offsite mirror, but that is cost prohibitive based on the size of my repository. It's VERY large. So instead, I archive to the cloud.

The post commit hook is fairly simple. All I do is write the revision number of the commit as a file on the server.   This is so the post commit hook can exit as quickly as possible, because of the aforementioned problem 1. Later at night, the server loops through the directory where all the days commits have been stored and runs a simple batch file on it.

@echo off
call setPath.bat
svnadmin dump E:\svndb -r %1 --incremental | bzip2 > svnarchive.%1.dump.bz 

:UPLOAD
echo "Upload dump!"
s3cmd put svnarchive.%1.dump.bz s3://svnarchive/svnarchive.%1.dump.bz

for /f "delims=" %%v in ('md5sum svnarchive.%1.dump.bz') do set TOOLOUTPUT=%%v
SET LOCAL_MD5=%TOOLOUTPUT:~0,32%
echo "Local md5 = %LOCAL_MD5%"

s3cmd info s3://svnarchive/svnarchive.%1.dump.bz | grep MD5 > rev%1.txt
set /p TOOLOUTPUT= < rev%1.txt
del rev%1.txt
SET REMOTE_MD5=%TOOLOUTPUT:~14,47%
echo "Remote md5 = %REMOTE_MD5%"
if "%LOCAL_MD5%" == "%REMOTE_MD5%" (
	echo "MATCH!"
	goto END
) else (
	echo "DOES NOT MATCH!"
	goto UPLOAD
)

:END
del svnarchive.%1.dump.bz

Simply put, this batch file takes the rev number as an argument, does an incremental dump and bz2's that output. Then it uploads it to S3, it then compares S3's calculated MD5 with the locally calculated MD5 and if they match, voila! Delete the dump and move on to the next revision. If not it tries it again and again, until it succeeds. Using the S3 calculated MD5 is a simple way to keep bandwidth moving in and out of S3 as low as possible. The advantages of using this method is that it is not always safe to make a copy of an svn repo while it is running, so simple copy backup solution on a repo is insufficient. You could do a svn hotcopy before running regular backup software on it, but to hotcopy a large repo could take a long time. This method of doing a per transaction backup is quick and cheap in comparison.


Additional Steps

 In addition to this, regularly run integrity checks are always a good idea. I can run an integrity check on any revision by redumping the revision and checking the MD5 against the one in S3. I re-verify the MD5 of every revision committed in any given week, at the end of the week. Yearly, I also do a full verification from 1 to N. If there are differences discovered, then it's time to check the mirror, and see which one is right. Because the corruption could have come from the main repo, or S3! In the event of a catastrophic failure that takes out both the main and the mirror server, I also have a recovery script that grabs every single revision from the bucket and does a svnadmin load to a new repo. Something like :

for /L %%V in (%1,1,%2) do (
	s3 get svnarchive/svnarchive.%%V.dump.bz /nogui
	bunzip2 svnarchive.%%V.dump.bz
	svnadmin load svndb < svnarchive.%%V.dump
	del svnarchive.%%V.dump
)


Limitations

Amazon has unlimited files number per amazon s3 bucket so theoretically this archival system could go forever, but S3 has a file size limit of 5GB. Anyone who commits 5GB at a time to SVN needs to be shot.

The cloud is not always safe, there have been a couple of reports of corruption and failures with the s3 service, but for the most part it has been a reliable service for me.

Restoring from S3 could take a while, depending on my downstream, but if BOTH mirror and main server went down. I think I have bigger problems, like the big one finally hit Southern California, or someone burned down the building.

This has worked fairly well for me for quite a while now, and I hope this gives someone else ideas on cost effective backups, redundancy and archival. I do know that people recommend having multiple repositories, but the way we have our projects set up, it's much easier to do them in one repo.