samba and big files question 7192

On Wed, 14 Dec 2005 09:30:37 +0000, Robert Latest

There are even more complications than that.

Caveats: I have no experience with weex. This response is long.

I agree with Stan's earlier response, if your data can served directly from its origin (your physical location), then there may be no need to sync at all- you could serve it for them. But it might still make sense to push it out to them. You need to study which approach is best. The first approach is better if your data will be used infrequently by the clients. The second approach is better if they are using it a lot. If they are using it a lot, then if you are serving it, there may be time lags where the end user is waiting for the data to be transferred. If the files are large, then lags are noticable even with DSL or T1. In any case, you should estimate how much time-bandwidth each transfer method would require using the given communication channel.

The remainder of this response buttumes that it is better for both parties to transmit the complete data set all at once. When it is in place, the updated set will be then made available on their network. See end comments for the summary of how I envision this working.

I would recommend that you investigate using rsync, possibly using ssh as the transport mechansim for security. BTW, you should also buttess the security requirements for your data while in transit.

Be aware that the same (or similar) problem crops up as you are having with ftp (weex) with rsync. File locks seem to be the tricky sticking point. Whatever method you choose needs to account for files which may be in use by users before they are changed (or yanked away) by your update procedures. How do you ensure that you haven't "yanked" away a file that someone is reading? I don't have the solution to that problem, but I have proposed a solution below which may work as a starting point. Below, I have begun an outline for one possible approach to this type of problem.

DISCLAIMER: There is NO WARRANTY or claim of suitability for a specific purpose for the method presented below. Verify all commands are correct and appropriate for your specific situation before proceeding. You are responsible for any commands you issue on your system. The scripts below have only rudimentary error checking- flesh out to meet your nees.

Again, this approach does not take into account file locking.

Here is the general outline. It buttume your server is running the ssh daemon, with an account accessible by your clients. Another option is for your data to be hosted by an rsync server. ssh is encrypted, which is a benefit if your data should not be transmitted "in the clear."

CAUTION: This method uses rsync, a command that can do great damage if used improperly. Experiment with directories where you can limit the damage until you have it down pat. It is picky about trailing slashes and dots on pathnames, so make sure you are clear about your intended source and destination before starting rsync. Obviously, the --delete option can be dangerous.

Outline

0. Create an appropriate "aging" scheme that meets the needs of you and your client. One approach is to use directories for the "last", "current", and "next" rsync cycles. "Aging" occurs when an rsync cycle completes. As soon as the "next" data is in place, then the following can occur:

a. Remove "last" directory. b. Move "current" to "last". c. Move "next" to "current"

1. The client logs into their local computer and prepares a new destination for the data they will pull from your server on this rsync cycle. a. Prune "last" directory (or as a separate operation when rsync is complete.) b. Create timestamped or other obvious directory naming for the "next" rsync target. c. Duplicate directory structure of existing data using "current" into "next". Here is a simple implementation:

#!-bin-bash # # Arguments: # 1. Parent directory for "last," "current," "next." # 2. name of "current" directory, relative to parent # 3. name of "next" directory, relative to parent # IFS=$'-n' $# -ne 3 && exit 1 !( -e $1 && -d $1 ) && exit 1 PARENT=$(cd $1 && pwd) echo $2 grep "^--" && exit 1 echo $3 grep "^--" && exit 1 SOURCE=${PARENT}-${2} TARGET=${PARENT}-${3} !( -e $SOURCE && -d $SOURCE ) && exit 1 -e $TARGET && exit 1 touch $TARGET ! -e $TARGET && exit 1 rm $TARGET for i in $(cd $SOURCE && find . -type d sort);do mkdir -p ${TARGET}-${i} done #end script

d. Duplicate files from "current" into "next" with hard links, man ln. Here is a simple implementation:

#!-bin-bash # # Arguments: # 1. Parent directory for "last," "current," "next." # 2. name of "current" directory, relative to parent # 3. name of "next" directory, relative to parent # IFS=$'-n' $# -ne 3 && exit 1 !( -e $1 && -d $1 ) && exit 1 PARENT=$(cd $1 && pwd) echo $2 grep "^--" && exit 1 echo $3 grep "^--" && exit 1 SOURCE=${PARENT}-${2} TARGET=${PARENT}-${3} !( -e $SOURCE && -d $SOURCE ) && exit 1 !( -e $TARGET && -d $TARGET ) && exit 1 for i in $(cd $SOURCE && find . -type f);do SF=${SOURCE}-${i} DF=${TARGET}-${i} ln $SF $DF done #end script

2. The client checks for updates regularly on your server. If changes are ready, they can update the contents of the "next" directory. Here is a simple implementation:

#!-bin-bash # # Arguments: # 1. Parent directory for "last," "current," "next." # 2. name of "next" directory, relative to parent # IFS=$'-n' $# -ne 2 && exit 1 !( -e $1 && -d $1 ) && exit 1 PARENT=$(cd $1 && pwd) echo $2 grep "^--" && exit 1

SOURCEIP=your.source.host SOURCEDIR=-forclientx-remotedatadir# use trailing slash in above line for in this specific situation, rsync usage TARGET=${PARENT}-${2} !( -e $TARGET && -d $TARGET ) && exit 1 # # pull using rsync over ssh transport # (cd $TARGET && rsync -lprtv --progress --delete ${SOURCEIP}:${SOURCEDIR} .) #end script

Summary Maybe, this will get you started if you decide to use an rsync method. It could work, and it could be entirely scripted. You would give your clients the means to access the data on your systems by giving them an account on your server, and the scripts (or master script) which they could use to keep current. IF you use ssh transport, then the clients will need an account on your system with ssh priviledge enabled. I recommend using ssh certificates to avoid the trouble with pbuttwords.

-- Douglas Mayne