login      my ps      about      contact  



home instructor-led on-demand! screencasts blogs technical staff


CraigBlog » Announcing HoboCopy

Announcing HoboCopy

CraigBlog

Pluralsight Blogs

Syndication

Recent Posts

Tags

View more

Navigation

Archives

For a while now, I've been unhappy with the state of my backup strategy. I have a simple script that runs ntbackup every night on a rotating, 14-day schedule: one full backup followed by six incremental backups, then repeat with a different target. In this way, I can always recover to any day in (at least) the last week.
 
So why am I unhappy? Two reasons:
 
  1. ntbackup isn't exactly robust - I've seen all sorts of failures, and restoring files is sort of flaky.
  2. ntbackup locks its information up in a proprietary format. I just want my files, dammit, not some .bks file.
 
Being a cheap bastard with decent programming skills, I started looking at what it would take to wire up my own solution. I thought about using robocopy, the king of copy tools. But it has one major drawback: it can't copy any files that are locked by another program. Some of my friends solve this problem by shutting down all their programs every night, but I knew that there was no way I (or my wife) was going to remember to do that, and of course the one time I'd forget was the day my hard drive would tank. Plus, what about programs like SQL Server that I don't want to ever shut down?
 
Obviously, it's possible to copy files that are in use. After all, ntbackup does it. So I started poking around, and came across VSS. The good VSS (Volume Shadow Service), not the unbelievably crappy one (Visual Source Safe). The Volume Shadow Service is a very cool piece of Windows XP/2003 that lets you "snapshot" a hard drive, creating what's essentially a point-in-time image of what's on the disk. You can then copy files from that image at your leisure. Better, it's done in an efficient manner, so that it doesn't actually copy anything unless someone does a write, and even then it only copies at a block level. Which is good, because otherwise you'd need 50GB free to snapshot 50GB of data.
 
But it's even better than that. VSS includes an API that programs like SQL Server 2005 can use to find out when a snapshot is about to occur. When so notified, VSS-aware programs can flush their state to disk, so you get a consistent backup. Of course, not every program is aware of VSS. For those, you get what the docs call a "crash consistent" snapshot. Translation: whatever the hell was on the disk. In my book, that's still better than not backing up at all. After all, my computer seems to do just fine after a BSOD, which is no different.
 
Armed with the Platform SDK docs, I set out to achieve my goal of a VSS-based backup utility. I would have liked to have used robocopy to do the copy, but I couldn't get it to copy using source paths like \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\data\backup\SmtpSend\Properties\Resources.resx,
which how you get to the files in the snapshot.
 
Next, I tried to write a managed tool that would do the copy. That wouldn't have been too bad - recursive file copy is pretty simple to implement. Unfortunately, the VSS API is completely broken for CLR interop - the main object you need to access implements one COM interface,  IVssBackupComponents. But if you try to query that object for that interface, it returns E_NOINTERFACE. Which is wrong, wrong, wrong. And also means that there's no way to use the object from straight managed code.
 
So I decided to write the tool in unmanaged C++. Maybe I could have done it in managed C++, or a combination of C++ and C#, but in the end I decided to make this a project that would help me get a little rust off my C++ skills. Plus, the documentation suggests that paths starting with \\?\GLOBALROOT aren't safe to use from managed code, although it worked fine when I tested it. The end result is HoboCopy. You pass it a source directory and a destination directory, and it makes a recursive copy using VSS. I've run it against my entire hard drive, and it's able to copy everything except files I didn't have permission to copy (e.g. some stuff under Documents and Settings), and for those situations I've got the /skipdenied switch. Some day I'll add a switch that enables the SE_BACKUP privilege to make that behavior even better. Unless one of you wants to submit a patch. :)
 
Speaking of patches. As of now, the project is hosted at SourceForge under the MIT license, so have at it! (Make sure you download the right one - there are separate binaries for XP and W2K3.) Just go easy on me if you look at the source - it's been years since I wrote much C++. :)
Posted Sep 20 2006, 06:54 AM by craig-andera

Comments

DuncanS wrote re: Announcing HoboCopy
on 09-20-2006 7:51 AM
Hmmnn, I got the interop to work, you use the VSSCoordinatorClass class:


VSS.VSSCoordinator vss = new VSS.VSSCoordinatorClass();

vss.StartSnapshotSet(out snapshotSetID);
vss.AddToSnapshotSet(this.VolumeName, Guid.Empty, out _snapshotID);

VSS.IVssAsync vssAync;
object callback = null;
vss.DoSnapshotSet(callback, out vssAync);
vss.DoSnapshotSet(callback, out vssAync);

while(true)
{
int hr, x=0;
vssAync.QueryStatus(out hr, ref x);
Console.Write(".");
if((AsyncStatus)hr == AsyncStatus.Finished)
break;
Thread.Sleep(1000);
}

VSS._VSS_SNAPSHOT_PROP snapshotProps;
vss.GetSnapshotProperties(_snapshotID, out snapshotProps);
_snapshotDeviceName = snapshotProps.m_pwszSnapshotDeviceObject;

...
Haacked wrote re: Announcing HoboCopy
on 09-20-2006 8:16 AM
Dang you're cheap. You couldn't just by Achronos?

Cool stuff though!
Haacked wrote re: Announcing HoboCopy
on 09-20-2006 8:25 AM
http://www.genie-soft.com/ does zip backup.
Craig wrote re: Announcing HoboCopy
on 09-20-2006 8:59 AM
Ah! Didn't know about the coordinator class...where were you two months ago!? :) Thanks for the heads-up, though. Maybe I'll switch it to work managed some day.

And yes, I am cheap. :)
Craig wrote re: Announcing HoboCopy
on 09-20-2006 9:05 AM
Wait - where did you find VSSCoordinator? I only see one pretty worthless hit on it on Google...
Craig wrote re: Announcing HoboCopy
on 09-20-2006 12:03 PM
OK, found it. It's right there in the registry. :)

I'm still not 100% sure this would work - as I said I found some sort of warning about using interop to copy files out of the \\?\GLOBALROOT namespace. But if it does work, it sure as hell would be easier than what I'm doing now.
Jason Haley wrote Interesting Finds: September 20, 2006
on 09-20-2006 7:31 PM
Eric Bowen wrote re: Announcing HoboCopy
on 09-21-2006 5:47 AM
You can mount a VSS Shadow Copy as a drive letter, and then you CAN use RoboCopy (I'll try to cook up a demo on my blog).
Craig wrote re: Announcing HoboCopy
on 09-21-2006 6:28 AM
Except you can't create persistent shadow copies on XP. Otherwise that would be a great solution.
Steve Floyd wrote re: Announcing HoboCopy
on 09-22-2006 7:56 AM
FileKeeper (http://www.filekeeper.com) has nailed the continuous backup problem to the wall. Really slick instantaneous backup with right-click recovery for all versions.

I hit ^S (Save) in Word and it grabs the difference and backs it up that instant. Really cool.

Volume Shadow Service is limited to so many copies.
John McPherson wrote re: Announcing HoboCopy
on 09-25-2006 5:02 AM
Cool stuff man, I also LIKE very much the fact that you code the starting { and ending } on seperate lines, this is the same tech. that I use and it makes C, C++, C# C whatever code so much more readable...

Thanks,
Craig wrote re: Announcing HoboCopy
on 09-25-2006 5:45 PM
Steve: FileKeeper sounds cool, but at "not free" it doesn't meet my cheapness requirements. :) As for VSS being limited, the only limitation I know of is for persistent copies, which HoboCopy doesn't use.

John: glad you liked the style - writing C++ after years and years of C# was a major challenge. As it is, I find the HoboCopy code to be a real mishmash of traditional C++ and C# styles. But given that there's a COM component that gives access to the VSS API, I'll probably rewrite HoboCopy as managed code the next time I need to make significant changes - it's just soooo much easier to get work done in C#.
Jmtyra wrote re: Announcing HoboCopy
on 11-30-2006 9:08 AM
LOVE Hobocopy, this is ~exactly~ what I've been looking for. I have been using some other solutions, but never been able to backup locked/in-use files.

Awesome job Craig, you da man!! =D
Craig wrote re: Announcing HoboCopy
on 11-30-2006 9:42 AM
Glad you like it. Be warned, I've found one major bug that I've got a pending fix for. Basically, the part where it tells SQL Server (or whatever other writers) that a snapshot is about to occur is wrong in the publicly released version. Straight file copying still works, but you're going to get a "crash consistent" view of files, not a "backup consistent" one. I'll be releasing a new version sometime in the next few weeks to correct this.
Ken wrote re: Announcing HoboCopy
on 11-30-2006 10:57 AM
TYVM... tis almost a perfect fix for my desire for a background backup of my notebook.

I've been getting an error message when I've tried to specify a remote unc (\\server\share) as the destination. Am I doing something wrong or is this a restriction in the current version?

Thanks again!
Craig wrote re: Announcing HoboCopy
on 11-30-2006 11:47 AM
No, that's a scenario I've explicitly supported. In fact, I had to do a little extra work to make sure it works.

What error are you getting? Can you run with /loglevel=4 and send me the log?
Ken wrote re: Announcing HoboCopy
on 12-01-2006 11:34 AM
The problem was a lack of patience on my part. I had aborted a local backup copy process, and then tried to start a new one to the UNC. I've a feeling (could be wrong!) that the VSS was still processing the first request, and so hobocopy gave me an error. After playing with it some more I managed to get it working.

On another 2 notebooks however, the story is different. When I try to run the program for them I get the following error:

"The system cannot execute the specified program."

It's running from the same network share for both machines, and I get the same error when I copy the hobofiles local to the c drive. Any ideas?

<bow> thanks!

Ken
kenlowe65
at gmail dot com.
Craig wrote re: Announcing HoboCopy
on 12-01-2006 12:09 PM
ah, yes, Hobocopy is not particularly good yet at recovering from aborted errors. I believe if you wait a while, VSS will clean up after itself. A reboot will probably also do it, as will obtaining a copy of vshadow.exe and running it with -da (delete all shadow copies).

As for the "system cannot execute the specified program" error, that's a bit weird. It sounds like something important is missing. Can you point a copy of depends.exe at the EXE and see if it shows any missing dependencies? Perhaps the C runtime?

The other thing I can think of is that it might be an OS/platform thing. Are you trying to run it on a 64 bit box, or using the XP version on W2K3/Vista or vice-versa?
Ken wrote re: Announcing HoboCopy
on 12-01-2006 6:41 PM
It has worked fine on two WinXP sp2 notebooks (HP Tablet and Dell D820). It has also failed on two WinXP sp2 notebooks (a Fujitsu tablet and Toshiba portege). Very strange.

I will try the "depends.exe" and see if there is any difference between the machines on Monday.
Ken wrote re: Announcing HoboCopy
on 12-04-2006 6:08 AM
Dependency failures:
MSVCR80.dll
MSJAVA.dll
Craig wrote re: Announcing HoboCopy
on 12-05-2006 3:50 AM
MSJAVA? WTF? I'm definitely not using Java. I think you can ignore that one.

MSVCR80.dll, however, is the C Runtime. I don't know what you installed on one machine versus the other that you got the CRT one place but not the other, but that's likely it.

A little googling shows that MSVCR80.dll can be downloaded. Give it a try and see if that helps. You should just be able to drop it in the same dir as Hobocopy.exe.
Chris Wilson wrote re: Announcing HoboCopy
on 12-06-2006 2:24 PM
Hi Craig,

Thanks for writing this and making it available under the MIT license! I'm working on a backup program (Box Backup) that I want to add VSS support to, and this should help if I need sample code that's compatible with the license (unlike the VSS SDK).

I'm trying to access the methods of IVssBackupComponents by IDispatch, rather than vtable, to help users who don't have the header files and want to compile the code. However, I think that maybe VSS doesn't support IDispatch properly.

I can get an IDispatch interface from IVssBackupComponents, but when I try to get the DISPID of the AbortBackup method using GetIDsOfNames, I get the error (HRESULT) 0x8002801D, which seems to be TYPE_E_LIBNOTREGISTERED.

This is not a documented return code from GetIDsOfNames. Any ideas what I can do, if anything?

Cheers, Chris.
Chris Wilson wrote re: Announcing HoboCopy
on 12-06-2006 2:48 PM
Hi Craig,

Also, if I try to get an IDispatch interface from VSS.VSSCoordinator, CoCreateInstance returns E_NOINTERFACE. Do you know how a scripting language such as C# manages to call methods on this interface? Is there a header provided, or some other way to call methods such as StartSnapshotSet, as DuncanS seems to have managed to do from C#? (I have never programmed in C#, I'm using C++ to try to get this interface).

Cheers, Chris.
Craig wrote re: Announcing HoboCopy
on 12-07-2006 3:31 AM
First, I looked into using VssCoordinator, but it doesn't support anything like the full VSS API, so I can't use it for Hobocopy. You may or may not encounter the same limitations.

Second, there's no reason at all that VSSCoordinator has to implement IDispatch - it's completely optional. In fact, it comes with a set of restrictions that may well have been too severe for them.

Third, C# isn't a scripting language - it doesn't have to use IDispatch at all (and in fact doesn't) when calling VSSCoordinator methods.

HTH.
Chris Wilson wrote re: Announcing HoboCopy
on 12-08-2006 12:20 AM
Hi Craig,

Thanks for your answers. If C# doesn't use IDispatch, do you know how it does call methods of e.g. VSSCoordinator? Presumably it doesn't understand C++ headers? Does it use the type library and LoadRegTypeLib, ITypeInfo and Invoke? Some other mechanism?

Cheers, Chris.
Craig wrote re: Announcing HoboCopy
on 12-08-2006 3:09 AM
It simply builds vtables, which after all are pretty simple structures. It can do this by importing type library information, but that's not necessary - as long as it knows the type and order of methods and parameters, it can do the right thing, just like any other COM client.
Jack wrote re: Announcing HoboCopy
on 12-27-2006 3:11 AM
I am having the same error as Ken. I have two windows 2003 servers. One of them hobocopy works and on the other it dosen't. In a batch file it says "The system can not execute the specified program." I downloaded the MSVCR80.dll but that didn't help. I downloaded the hobocopy-0.6.0.0-w2k3-x32-release.zip. The depends had two errors, msjava.dll and msvcr80.dll. the errors in depends are gone now after downloading those two files but the program still has the same error.
Craig wrote re: Announcing HoboCopy
on 12-27-2006 5:45 AM
OK, instead of going at the DLLs piecemeal, I should have suggested installing the Visual C++ 2005 redistributable package, available here:

http://www.microsoft.com/downloads/details.aspx?familyid=32BC1BEE-A3F9-4C13-9C99-220B62A191EE&displaylang=en

Give that a shot and see if it fixes the problem. If it does, I'll start shipping it with HoboCopy. I have a new version packaged up and ready to go that I'll push out the door before too long.
Jack wrote re: Announcing HoboCopy
on 12-28-2006 2:36 AM
Wow that did fix it, thanks for all your work and a great program!
CraigBlog wrote Announcing HoboCopy 1.0.0.0
on 01-08-2007 5:36 AM
Jan wrote Run command as soon as the snapshot has been taken
on 10-05-2007 6:24 AM
Hi,

HoboCopy seems to be exactly the tool I've been looking for. I intend to use it to backup VMware virtual machine files. In order to get the VM in a consistent state, I need to suspend (pause) it and make a copy of that paused state.

I would like to use HoboCopy with its VSS capabilities to minimize downtime, i.e.

1. Suspend VM
2. Take Snapshot
3. Resume VM as soon as snapshot has been taken
4. Do the actual backup

Is there any way to know when VSS is done with taking the snapshot?

Thanks for the great app,
Jan
Craig wrote re: Announcing HoboCopy
on 10-06-2007 4:16 AM
I don't know of any way to do manually what you're asking. I suppose the easiest thing would be to modify Hobocopy so that you can give it a script that it runs before/after the snapshot. I'm not planning to spend any time on Hobocopy to do that right now, but you have the source and it shouldn't be a difficult modification.

Ideally, the way this would work is that VMWare would provide hooks into the VSS APIs that would take care of this automatically. VSS is designed specifically to allow for the exact case you describe (this is similar, for example, to what SQL Server does when you take a shadow copy) but apparently VMWare doesn't support it right now. Too bad. But when it does, hobocopy should "just work" the way you want it to.
Jan wrote re: Announcing HoboCopy
on 10-06-2007 7:38 AM
Hi Craig,

thanks for your hints on implementing it. My C++ skills are barely existent, but maybe I'll ask a friend to do it.

For why VMware Server (which I use) doesn't natively support VSS: I guess it's marketing strategy. "Server" is their free product, designed to get people hooked on virtualization. Their enterprise product, ESX, which costs quite a few Dollars, has, afair, advanced snapshot capablities (in combination with their own prorietary file system, VMFS) and there are a number of products availabled specifically designed to do live backups of VMs.

Cheers, Jan
Lee wrote re: Announcing HoboCopy
on 11-15-2007 1:14 PM
Hi,

i'm very interesting by your works, but i can't find VSSCoordinatorClass,

can you help me ?

Thanks
Craig wrote re: Announcing HoboCopy
on 11-16-2007 2:50 AM
I'm not sure where it comes from. Probably when you install the VSS SDK.
Lee wrote re: Announcing HoboCopy
on 11-16-2007 2:58 PM
I have already installed VSS SDK, but i don't know how add a reference to VSS in a .net project (i have no VSS class available in .net or com reference list).

Can u tell me where are u referenced the class in your .net project ?

sorry for my poor english language ...
Craig wrote re: Announcing HoboCopy
on 11-18-2007 3:13 AM
Actually, I'm not sure...just went looking for it, and couldn't track it down. I think it's probably a managed DLL that ships with the SDK, but I can't find it now. Sorry: been quite a while since I worked on this stuff.
Ned wrote re: Announcing HoboCopy
on 11-19-2007 10:53 PM
Hi Craig,

First - great tool! I'm trying to use this for a backup solution, and I can almost use this to reconstruct a bootable windoze HD. What I do is run Hobocopy on the entire HD (C:) to a network share. Then, I plug a new (empty) hd back in, boot off of a CD, format and copy the boot sector, mount the network share, and proceed to copy over everything to the "new" HD. When I boot, I get a number of "strange" errors. It does boot into windows, but the system is unstable and doesn't seem to work correctly. I checked the backup log, and it seems to have aborted and skipped all files in C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys, with Message "Error accessing File...Skipping". I'm also not entirely certain that the registry is copied over 100% intact. You mentioned that you use this yourself for your backups. Have you solved these problems? If so, can you share that? Thanks!
Craig wrote re: Announcing HoboCopy
on 11-20-2007 5:18 AM
I do use the tool myself, every night, on three different machines. But I don't try to back up the entire HD. It certainly seems like you're running into a simple security issue: just run Hobocopy as a user with sufficient privilege to copy all those files. Perhaps Local System?

As for the registry issue, I don't have any information on that - all Hobocopy does is create a shadow copy and then call CopyFile, so I'm not sure what it could do differently to correct any issues there...
Lee wrote re: Announcing HoboCopy
on 11-26-2007 1:12 PM
Hi Craig,

Can u tell me why Hobocopy can't make a shadow copy on one file. When i give just one file as a source parameter, hobocopy create a shadow copy because my disk is in activity. But i don't see the file in destination directory.

Thanks
Craig wrote re: Announcing HoboCopy
on 11-26-2007 4:07 PM
The usage is:

hobocopy sourcedir destdir filename

So you always have to specify two directories: the directory to copy from and the directory to copy to. You can restrict it to just one file by specifying the filename.
Lee wrote re: Announcing HoboCopy
on 11-26-2007 11:29 PM
Thanks a lot Craig, i have missing a line in your documentation ...
Rob wrote re: Announcing HoboCopy
on 12-05-2007 10:01 AM
I noticed if you try and copy a file to a unc path it the tool creates the directory structure where ever your cmd prompt location is set to. i.e. by default in xp the cmd prompt is c:\documents and settings\%username%, if you look in that directory you'll see the unc directory structure. if the dest is a drive letter it's all good.
Rob wrote re: Announcing HoboCopy
on 12-05-2007 10:03 AM
hobocopy is great though, nice job
Craig wrote re: Announcing HoboCopy
on 12-05-2007 10:36 AM
Sounds like a bug. I'll file it and if I ever get back to making improvements...
wajahat wrote re: Announcing HoboCopy
on 12-28-2007 4:09 AM
I am unable to find atlstr.h which is used in Hobocopy . Can anyone help me i will be very thankfull
Craig wrote re: Announcing HoboCopy
on 12-28-2007 4:49 AM
On my machine, it's in my VS2005 directory (for me, that's C:\program files\vs2005 - yours will be different), under the subdirectory VC\atlmfc\include.
joeqi wrote re: Announcing HoboCopy
on 05-23-2008 1:36 AM
Hi, Craig ,
Your software is one of the best choices to my need.

but unfortunately, i met a problem.
I am trying to backup all the data in my c: partition to my e: partition. i login as an administrator and run "HoboCopy.exe /r c:\ e:\" in the console, but the result is always:
"Creation of directory failed with error Access is denied. (Error number 5) on directory \\?\e:\"

but if i run "HoboCopy.exe /r c:\ e:\1\" , everything is ok?

any solution?

joeqi wrote re: Announcing HoboCopy
on 05-23-2008 1:44 AM
but if i run "HoboCopy.exe /r c:\ e:\1\" , everything is ok...
Craig wrote re: Announcing HoboCopy
on 05-23-2008 2:00 AM
Error 5 is a permission problem. It sounds like Hobocopy is unable to create the directories it needs to on the E: drive. That may well be a bug in Hobocopy - possibly it's barfing when trying to create the root directory, although IIRC it shouldn't try to create the directory if it already exists.

I think you've hit on the workaround for now - don't use a root directory as the destination. One thing you could try would be to use linkd to "mount" your E: drive at some other location and then use that as the destination. E.g.:

linkd D:\mount\e E:\

hobocopy /r C:\ D:\mount\e

I'll investigate the bug, but it'll probably be a while before I actually get around to fixing it. Sounds like you have a workaround in the meantime.
joeqi wrote re: Announcing HoboCopy
on 05-23-2008 2:38 AM
thx, Craig. I hope that hobocopy will become better and better.
Jonas Lewin wrote Hobocopy would solve many of my problems.
on 06-17-2008 11:31 AM
Hello. I am desperately looking for the same sollution as you describe in your blog:
Robocopy with the shadow copy ability.
I only have one difference. I only run Vista x64 computers.

This is the result of:
hobocopy c:\mail c:\backup outlook.pst
Starting a full copy from c:\mail to c:\backup
There was a COM failure 0x8000ffff - .\HoboCopy.cpp (172)

Is there an easy solution available, other than rewriting the code in managed .net, as I know that is a quite big challange?

/Jonas
Craig wrote re: Announcing HoboCopy
on 06-17-2008 11:35 AM
I just need to recompile it for x64. As I've recently upgraded to a 64-bit machine myself, this is something I plan to do before too long.
Jonas Lewin wrote Thanks
on 06-18-2008 8:52 AM
Hello.
Thanks. This sounds great. I would be happy to be a beta tester :-) Please send me an email when you got the 64 bit version to work.
/Jonas
Craig wrote re: Announcing HoboCopy
on 06-25-2008 6:41 AM
The x64 version is now available on the SourceForge download page.
piperfect wrote re: Announcing HoboCopy
on 06-27-2008 5:17 PM

YOU ROCK!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

i needed shadow copy and hate ntbackup....

perfect solution for me.

Thomas Goddard wrote re: Announcing HoboCopy
on 07-08-2008 7:07 PM

Yes, you do indeed rock for creating this.  They removed support for file copy in Windows Server 2008 in an attempt to monopolize the backup market with Microsoft Data Protection manager.  Thank you so much for saving me.

craig-andera wrote re: Announcing HoboCopy
on 07-09-2008 5:22 AM

I'm glad you find it helpful. But what do you mean by "The removed support for file copy"?

Vinoth Sakthivel wrote re: Announcing HoboCopy
on 07-13-2008 8:54 AM

hi,

i am getting this error

Starting a full copy from X:\ to d:\

There was a COM failure 0x80042308 - .\HoboCopy.cpp (349)

can u pls give me asolution

craig-andera wrote re: Announcing HoboCopy
on 07-13-2008 2:12 PM

What sort of drive is X: ? You can't use hobocopy to copy from a networked/mapped drive, because you can't get the remote machine to make a shadow copy.

Vinoth Sakthivel wrote re: Announcing HoboCopy
on 07-14-2008 12:03 AM

Thank you, for your quick response. can you please advise me how to take backup of open files in the network?

craig-andera wrote re: Announcing HoboCopy
on 07-14-2008 6:05 AM

Run hobocopy on the machine where the files live. Other than that, I'm not sure how you'd do it. Maybe some commerical backup utilities have the ability - no idea.

val wrote re: Announcing HoboCopy
on 08-12-2008 12:33 AM

Thanx for creating such a great app.

Just one question though.

Does Hobocopy support long directory paths or directory paths with spaces in their names?

I'm getting the error below trying to back up my Outlook 2003 PST file from the default installation directory..

I'm using the standard windows quotes ("") to enclose the full directory path to solve this as without them Hobocopy interprets directory paths to the first space as a command argument

Hobocopy works like a dream in all my other test cases.

See sample input-output below:  

C:\Temp\software\Hobocopy\HoboCopy-1.0.0.0-XP-32bit-Release>hobocopy /full /skipdenied /y /r "C:\Documents and Settings\Administrator\Local Settings\Application Data\Microsoft\Outlook\" C:\Temp\testbak2\

HoboCopy (c) 2006 Wangdera Corporation. hobocopy@wangdera.com

Error calling GetFullPathName: The parameter is incorrect.

(Error number 87)

C:\Temp\software\Hobocopy\HoboCopy-1.0.0.0-XP-32bit-Release>

I have XP 32 bit and have backed up other data with Hobocopy to the same destination folder.

Any ideas?

Thanx.

craig-andera wrote re: Announcing HoboCopy
on 08-12-2008 5:11 AM

Huh - I can totally repro that. I'll take a look. Probably I'm not handling long pathnames correctly somehow.

Workaround: create a link to that directory with linkd and back that up.

linkd C:\temp\hobocopy-test "C:\Documents and Settings\Administrator\Local Settings\Application Data\Microsoft\Outlook\"

hobocopy /full /skipdenied /y /r C:\temp\hobocopy-test C:\Temp\testbak2\

Creating Snapshot Style Incremental Rotating Backups in Windows | Alpha Leonis wrote Creating Snapshot Style Incremental Rotating Backups in Windows | Alpha Leonis
on 08-12-2008 10:32 AM

Pingback from  Creating Snapshot Style Incremental Rotating Backups in Windows | Alpha Leonis

AlphaVSS - Bringing Windows Shadow Copy Service (VSS) to .NET | Alpha Leonis wrote AlphaVSS - Bringing Windows Shadow Copy Service (VSS) to .NET | Alpha Leonis
on 08-12-2008 11:40 AM

Pingback from  AlphaVSS - Bringing Windows Shadow Copy Service (VSS) to .NET | Alpha Leonis

Viru wrote re: Announcing HoboCopy
on 09-12-2008 1:11 AM

Hi Craig,

Is it possible to create a persistent shadow copy in XP (or Win2008)? I tried to run betest.exe supplied with VSS SDK on XP and Win2008. On XP it could not run it, while on Win2008 it runs fine, but where can I find the created shadow copy?

Any help would be appreciated.

Thanks,

Viru

craig-andera wrote re: Announcing HoboCopy
on 09-12-2008 6:09 AM

No, it's not possible. Otherwise I would have used something like vshadow to create and mount the shadow copy and then used robocopy to copy the files. As soon as the process creating the shadow copy ends, the shadow copy goes away.

Vshadow does have an option for running a program before terminating, so you might be able to use that, but I'm not sure whether you're allowed to mount the shadow copy on a drive letter, so actually accessing it might be hard.

Viru wrote re: Announcing HoboCopy
on 09-15-2008 8:13 AM

Hi Craig,

Thanks for the reply.

I tried to modify your code by adding a call to SetContext(VSS_CTX_NAS_ROLLBACK) and commenting out writers involvement + code for backup. After I run your modified code, I could see vshadow returning shadow copy (created by your code) with -q option.

Thus it seems VSS does support persistent shadow copies. Actually what I am trying to achieve is that I want to use a NDMP compliant backup software & NDMP server for backup and recovery. So actual backup will be taken care by NDMP server and I just want to use VSS for creating snapshots.

Your comments on this will be helpful.

craig-andera wrote re: Announcing HoboCopy
on 09-15-2008 8:35 AM

Interesting! One question I would have is whether the shadow copy sticks around for long - I've seen cases where a shadow copy will survive after the program crashes, but it eventually gets cleaned up. I'm not sure how often that happens - certainly a reboot will do it.

Other than that, I'm not sure what advice to offer. At this point you've done more research on it than I have. :)

I agree that a tool that just creates the persistent snapshot would be more useful than hobocopy, as robocopy already provides far better copying utility than hobocopy. So if you can figure out how to get a drive mapped to a persistent shadow copy under all operating systems, that would be great!

Of course, I'm left to wonder: if it's possible, why can't vshadow do it? Have you looked at the vshadow source to see what it does?

Viru wrote re: Announcing HoboCopy
on 09-15-2008 11:53 PM

Craig, the shadow copies created by modified hobocopy does not get cleaned up after a reboot. It is there at least from the last couple of days.

after your advice, I tried to look into vshadow and I found out that there is a -el option to expose a shadow copy as drive or mount point. I tried to run vshadow with this option, but it crashed. I will look into vshadow code and try to find out what is making ti crash.

Viru wrote re: Announcing HoboCopy
on 09-16-2008 7:27 AM

Hi Craig, I could now mount snapshot to a (unused) drive letter or as a mount point using vshadow. This was exactly what I was trying to achieve.

But now, I have another doubt. Do you have any idea what role does SetBackupState play in creation of snapshot. I mean, if I call SetBackupState with (VSS_BT_FULL or VSS_BT_INCREMENTAL or VSS_BT_DIFFERENTIAL) does snapshot that will be created differ?

I have to support full, incremental and differential backups in my application. calling SetBackupState with appropriate backup type is all that I need to do? Or do I have to take care of these while actually backing up data?

Any comment on this?

craig-andera wrote re: Announcing HoboCopy
on 09-16-2008 7:50 AM

I have no idea what that flag actually does. It's possible that it's just passed through to the writers as an optimization, so they can determine what sort of data they need to flush to disk. For the file provider, it may make no difference at all.

But honestly, I'm surprised what you've done works at all. :)

Viru wrote re: Announcing HoboCopy
on 09-16-2008 8:08 AM

To be honest I haven't done anything. You can yourself confirm whatever I have done.

There's a flag -p in vshadow copy which can be used to create persistent shadow copies. Using this flag you can create a persistent shadow copy. (e.g. C:\vshadow -p D:). This will not cleaned up unless you call DeleteSnapshots

After creating a persistent shadow copy this may, you can get the snapshotID of this shadow copy using vshadow -q.

Now using vshadow -el={SnapID}, G: you can mount that shadow copy to G: drive of your system. You can get exactly same data in G: drive which was there on D: drive.

But you need windows server to run this. since winXP does not support persistent shadow copies.

craig-andera wrote re: Announcing HoboCopy
on 09-16-2008 8:13 AM

Well that explains my confusion: I thought you were doing all of this on Windows XP/Vista.

So it sounds like hobocopy is still needed, as not many people run a server OS. I have machines with both, and I wanted one tool that I could set up a uniform backup with.

Looking back over your comments, I see now that you were asking about persistent backups on XP *or* 2008. Sorry I didn't read more carefully, or I could have saved you a lot of time!

Viru wrote re: Announcing HoboCopy
on 09-16-2008 8:23 AM

Hobocopy is an excellent work Craig. And after talking to you only I looked at vshadow more carefully and found out this. So your help is really appreciated ... :)

felix wrote re: Announcing HoboCopy
on 09-18-2008 11:07 AM

doest hobocopy not work with server 2k3.  i try to use hobocopy on server 2003, this is the command i use " hobocopy /full <souce> <dest>. but it said " starting a full copy from <souce> to <dest> but nothing happen.

craig-andera wrote re: Announcing HoboCopy
on 09-18-2008 1:03 PM

Some people have reported trouble getting hobocopy working on 2003. I do have a 2003 machine that I run it on every night, though.

You can try running

hobocopy /verbosity=4 /full <source> <dest>

instead, and it should tell you a bit more about what's happening.

felix wrote re: Announcing HoboCopy
on 09-18-2008 5:13 PM

i got this with that command.

Calling CoInitialize

Starting a full copy from h:\quickbook\ to h:\quickbookbackup\

Calling CreateVssBackupComponents

Calling InitializeForBackup

Calling GatherWriterMetadata

c:\hobocopy\

craig-andera wrote re: Announcing HoboCopy
on 09-18-2008 8:35 PM

I assume H: is a network drive? Hobocopy doesn't work if the source is a network drive.

felix wrote re: Announcing HoboCopy
on 09-18-2008 9:41 PM

no h: is not a network drive

craig-andera wrote re: Announcing HoboCopy
on 09-19-2008 8:34 AM

OK, well truthfully I'm not really sure what to tell you, then. I did hear from someone else who was unable to make it work on 2003, but I can't reproduce that here.

Do you have a VM setup where you could try on a fresh install of 2003?

Mark wrote re: Announcing HoboCopy
on 09-21-2008 4:26 PM

After reading this article I started some digging and found out that you can use VSS snapshots with robocopy not only using Windows 2003 but also Windows XP:

www.eggheadcafe.com/.../workstation-open-file-bac.aspx

craig-andera wrote re: Announcing HoboCopy
on 09-22-2008 12:28 PM

Yeah, I actually considered doing something like that, but it seems a bit fragile to me somehow. That's probably just paranoia.

But should I ever have the time, it would probably make sense to rewrite hobocopy to mount the shadow copy on an unused drive letter and hand off the hard work to robocopy. That should make it much more robust and full-featured.

Kevin wrote re: Announcing HoboCopy
on 11-03-2008 8:07 PM

Thanks for the great tool, it was exactly what I was looking for.

I am using HoboCopy to copy the "Users" directory on Vista64.  The full copy works, but an incremental copy fails on some files in the "All Users/Microsoft" folder.  I get the following message:

"Error was The process cannot access the file because it is being used by another process.

(Error number 32)."

As a workaround, I just ignore this folder for incremental copies.  Any ideas?

Thanks again,

Kevin

craig-andera wrote re: Announcing HoboCopy
on 11-04-2008 4:45 AM

Huh. That's kind of weird - error 32 is exactly what hobocopy is supposed to prevent. I've never seen this myself, and can't imagine what would be causing it. And I use incremental copies all the time (six nights a week on four different computers, in fact).

There is a bug in hobocopy having to do with spaces in filenames, but it doesn't look like you're hitting it given that you can do a full copy - it should be the same code path.

So sorry, I don't really have any idea.

Kevin wrote re: Announcing HoboCopy
on 11-04-2008 10:31 AM

Thanks Craig for the quick reply.  It looks like it has something to do with the Windows Search service that isn't playing nice with VSS.  Shouldn't MS play nice with itself?  If I stop the service, run HoboCopy, and then start the service again everything works fine.  Maybe someone else will have the same problem...

With verbosity=4 I get the following output.

Skipping file All Users\Microsoft\Search\Data\Applications\Windows\GatherLogs\SystemIndex\SystemIndex.99.Crwl because it doesn't meet filter criteria.

Copied directory All Users\Microsoft\Search\Data\Applications\Windows\GatherLogs\SystemIndex

Copied directory All Users\Microsoft\Search\Data\Applications\Windows\GatherLogs

Copied file \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy73\Users\All Users\Microsoft\Search\Data\Applications\Windows\MSS.chk to \\?\Y:\Backup\McVista64\files\Users\All Users\Microsoft\Search\Data\Applications\Windows\MSS.chk

Unable to open file \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy73\Users\All Users\Microsoft\Search\Data\Applications\Windows\MSS.log exists. Error The process cannot access the file because it is being used by another process.

(Error number 32).

craig-andera wrote re: Announcing HoboCopy
on 11-07-2008 2:10 PM

Really weird. It almost looks like the search service is mounting the shadow copy and trying to index it. But that's just a guess.

Jens wrote re: Announcing HoboCopy
on 11-13-2008 3:10 PM

Great tool Craig! I detected this a couple of days ago and now I am adapting your nicely written VSS code to my open source program. It would have taken me weeks to come to this level! I am working on a  disk imaging program which is a nice counterpart to your work. Please note that the VSS integration is not available for download now, it is just work in progress. It will hopefully be soon. I am currently struggling with providing support for all paltforms in one binary. If you are interested in this please let me know. Your code is a much simpler and better sample than those in VSS SDK.

craig-andera wrote re: Announcing HoboCopy
on 11-13-2008 3:28 PM

Glad you like it. Shoot me a copy or a link of your thing when you get it done.

craig-andera wrote re: Announcing HoboCopy
on 11-13-2008 3:29 PM

@Kevin: I had this exact same problem the other day. I shut down Windows Defender and Windows Search and the problem went away. Not sure if either of those services was causing a problem or what.

Rick wrote re: Announcing HoboCopy
on 01-05-2009 11:06 PM

I have the same problem Felix had back in Sept.  Namely, when I run hobocopy on Server 2003 (sp2) I get:

C:\Documents and Settings\Administrator>\hobocopy /full /verbosity=4 d:\intuit\q

uickbooks c:\backup.place\quickbooks *.qbw

HoboCopy (c) 2006 Wangdera Corporation. hobocopy@wangdera.com

Calling CoInitialize

Starting a full copy from d:\intuit\quickbooks to c:\backup.place\quickbooks

Calling CreateVssBackupComponents

Calling InitializeForBackup

Calling GatherWriterMetadata

C:\Documents and Settings\Administrator>

Did you ever find any clues as to why it just stops at this point?

Thanks

craig-andera wrote re: Announcing HoboCopy
on 01-06-2009 6:18 AM

Sorry, no. Haven't had time to work on Hobocopy in a long time. I will say that I run hobocopy on my W2K3 box all the time without trouble.

Robert wrote re: Announcing HoboCopy
on 01-11-2009 6:55 PM

>>"The system can not execute the specified program."

I've seen this error as well on Windows 2003 Server x64 and it actually led me to this page. The problem is revealed by Dependency Walker as missing MSVCP90.dll and MSVCR90.dll.

Those files are not included in the redistributable for VC ++ 2005. The simpliest solution seems to be using the 2008 redistributable instead.

Hope this helps someone else.

Jens wrote re: Announcing HoboCopy
on 01-20-2009 12:46 PM

Craig, a couple of weeks ago I mentioned here that I adapted my code of the open source disk imager ODIN to use your (adapted) VSS code. I mentioned that I found a way to release only one binary that both supports XP and 2003, Vista. It took me a while I got so far to release a beta. If you are interested in the source code you can download it from here: sourceforge.net/.../platformdownload.php. It's embarassing how MS hast botched this API and made it incompatibel. So the solution is not nice and requires duplication of header information and structures. Anyway (for me) more convenient than to build and release two different binaries. In case of questions you can contact me through the sourceforge project page.

craig-andera wrote re: Announcing HoboCopy
on 01-21-2009 11:42 AM

Cool, thanks.

Matt wrote re: Announcing HoboCopy
on 02-02-2009 12:26 PM

Craig and to all.  This is a very interesting piece of code!  I am a programmer myself and had a problem to solve at work involving using Microsoft's ImageX tool to make a backup of a live Windows workstation (XP) or Windows 2003 servers and write this to a working .WIM file for system recovery purposes.  I originally started toying with the idea of writing my own VSS "wrapper" program that would allow ImageX, actually, may be my own implementation using the Wimgapi library, to backup locked/in-use files.  I then stumbled upon this site, looked over the code and decided that before I tackle a write of my own, did Microsoft provide some form of utility already that meets my goals?  There is no need to re-invent a wheel, unless I can make a better wheel:)  At any rate, here is what I have found:

Use Microsoft's own vshadow.exe located here: www.microsoft.com/.../details.aspx

Use dosdev.exe to map a drive letter to a mounted snapshot

Look at this site: www.eggheadcafe.com/.../workstation-open-file-bac.aspx

In my testing on a Dell Latitude D510 laptop running Windows XP SP2, I was able to backup the entire C: drive into a Windows .WIM file using Microsft's ImageX tool.  I then rebooted into Windows PE 2.0 on a USB pen drive, formatted the entire C: drive, then re-applied the Windows .WIM file back onto the empty C: drive, after which I restarted the machine.  Voila!  It works!  I was able to backup a live running instance of Windows XP SP2 into an ImageX .WIM file and then use this to completely recover the system.  I will run some tests on our Windows 2003 servers.

Note: Windows 2008 server comes with a program called: diskshadow.exe that is basically vshadow.exe and diskpart.exe rolled-up into one.  I am going to try that utility on Windows 2003 server and Windows XP to see if it works.

Steve wrote re: Announcing HoboCopy
on 02-11-2009 7:12 PM

Hi Craig and others,

I was looking for a way to make a duplicate boot drive copy in order to do some experiments with beta drivers and BIOS changes that (I discovered the hard way) tend to corrupt the boot drive.

After trying many different things (including Hobocopy) I stumbled upon some things that might be of interest to you and others that are now running Vista Ultimate.

I discovered that Vista creates a persistent shadow copy whenever you create a restore point (right click on "computer", select "properties", "advanced system settings", "system protection", and click "create").

Now if you open "computer" and right click on the drive you want to work with you can select "restore previous versions".  This doesn't actually restore anything yet and in here you can "open" the persistent shadow copy of the drive.  You can now do anything you like with this shadow copy, but more interesting is the path used to access the persistent shadow copy (it will be something like "\\localhost\C$\@GMT-2009.02.11-08.00.18") works with any application or utility (including robocopy, xcopy, dir, etc.).

For example you could type "robocopy \\localhost\C$\@GMT-2009.02.11-08.00.18 x:\ /e /b /copyall /purge /sl /r:5 /V /np /xj /tee /log:robocopy-c2x.txt" and this will make a complete, bootable copy of the entire C-drive on the X-drive.  By editing the registry (load hive) found on the X-drive to swap the C: and X: letters around and using "easyBCD" to add booting to X:, you end up with a fully bootable copy of the C-drive on the X-drive.  When you boot the X-drive, it now comes up as C and the original C remains safely unaltered (you can even unmount it so it is better protected).

So - in summary to access a shadow copy in Vista Ultimate (possibly other versions also):

1.  Create a "system restore point" which actually creates a persistent shadow copy.

2.  Open the latest "previous version" of the drive you want to work with and note the path (eg.  \\localhost\C$\@GMT-2009.02.11-08.00.18).  This actually opens the shadow copy in explorer.

3.  Note the path and use it with any application or utility, including robocopy, etc.  The path is persistent (you don't need to keep explorer open).

Note you can also create a shadow copy in Vista using the vssadmin command (instead of creating a restore point).  Then using "vssadmin list shadows" you can get the \\?\... path, but not all applications/utilities can access the shadow copy this way.  You can still use the "previous versions" trick to find the \\local\... path after using vssadmin to create a shadow copy.

Another trick I stumbled upon that also works well, but requires downloading the utility "dosdev.exe" from microsoft.com (download MPSRPT_CLUSTER.EXE and use winRAR to extract just dosdev.exe) is to do the following:

1.  Create a persistent shadow copy using "vssadmin create shadow /for=c:".

2.  Use "vssadmin list shadows" to find the \\?\... name of the shadow copy you want to work with.

3.  Use "dosdev T: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy4" (substitute the correct name) to mount the shadow copy temporarily to T:

4.  Now use any utility, including robocopy to work with the T-drive (such as robocopy T: X:...).  The only downside is T: can only be accessed from the command shell where you ran dosdev.exe, but this method works well for a batch script.

Note you can download MPSRPT_CLUSTER.EXE from www.microsoft.com/.../details.aspx (scroll down and download just MPSRPT_CLUSTER.EXE).  Dosdev.exe can be extracted using winRAR as I mentioned above and placed in any directory you wish (ideally in your PATH somewhere).

I hope this information helps some of you.  Hopefully these same commands will work with Windows 7 too.

Steve

marc wrote re: Announcing HoboCopy
on 04-14-2009 8:31 AM

I'm having a problem with the tool crashing:  I'm running on Vista and downloaded the 1.0.0.0-W2K3-Vista-32bit version.  Can anyone help out with this?   Thanks!

 Problem Event Name: APPCRASH

 Application Name: HoboCopy.exe

 Application Version: 0.0.0.0

 Application Timestamp: 459ea97d

 Fault Module Name: VSSAPI.DLL

 Fault Module Version: 6.0.6001.18000

 Fault Module Timestamp: 4791a76b

 Exception Code: c0000005

 Exception Offset: 0002bdcb

 OS Version: 6.0.6001.2.1.0.256.4

 Locale ID: 1033

 Additional Information 1: 78fe

 Additional Information 2: aa045f7c1ce52cf9005c58223248d7ac

 Additional Information 3: 9010

 Additional Information 4: 3b5117e91a8b8f54ca688ec720404d98

Add a Comment

(required)  
(optional)
(required)  
Remember Me?


home instructor-led training online training blogs

781.749.9238 (East Coast) | 310.251.9901 (West Coast) | Pluralsight, LLC , 185 Lincoln St., Suite 305, Hingham, MA 02043-1741

Copyright © 2008 Pluralsight LLC. All rights reserved.

designed by nukeation