IPCPullFile Function

Navigation: Scripting Reference Manual > Functions & Subroutines > IPCPullFile

IPCPullFile

Description

This function pulls (copies) the given file or files, specified in sourceFileSpec from each active remote node to the given destination destinationDirectory, which typically resides on the master machine. During the pull operation, a unique suffix will be appended to each file name copied to the destination in order to distinguish it from all the other files with the same file name that were pulled from other remote nodes.

The actual file copy occurs in two steps. The first step is to copy the file(s) from the source to the intermediate file transfer directory specified in the remote node's connection. To do this, the master sends a command to the remote node to copy the files and then waits until the remote reports that it has finished the file copy. In the second step, the master copies the file(s) from the intermediate transfer directory to the destination directory.

The names of the files successfully copied to the destination directory are placed in the files argument. This is very useful because the names of the files in the destination directory are usually not known (they have been renamed using a suffix unique to each remote) and because it may not be known how many files were successfully copied.

We recommend the use of the IPCWaitForIdle command prior the IPCPullFile to make sure all scripts have finished execution. Using IPCPullFile before the files being requested are generated by the remote script(s) may result in, (a) lengthy timeout scenarios while FRED attempts to retrieve the files (which do not exist), or (b) orphaned remote instances if the master script requires successfully pulled files before calling IPCTerminate to appropriately close the connections between master and remotes. If scenario (b) occurs (i.e. master script unintentionally fails to proceed to IPCTerminate), the orphaned remote FRED instances may need to be closed manually or be reconnected to through IPCReconnect for proper termination.

The text output of the command lists how many source files were found, how many were successfully copied to the intermediate transfer directory, and how many were successfully copied from the intermediate directory to the destination directory.

Syntax

count = IPCPullFile( connections(), sourceFileSpec, destinationDirectory, options, pulledFiles() )

Parameters

count (Long)

Returned number of files that were successfully pulled to the destination directory.

connections() As T_IPCINSTANCE

An array of T_IPCINSTANCE structures, with each structure defining the connection information for a remote node.

sourceFileSpec As String

The name of the file that will be pulled from the remote nodes (may include wildcard characters, resulting in multiple files being pulled). If the full path name of the file is not specified, the working directory each remote node will be assumed.

destinationDirectory As String

Full path to the directory where sourceFileSpec should be pulled (typically on the master node).

options As String

Space delimited string specifying options for file overwrite and move behavior, with the default empty string indicating "overwrite copy". Options are the following:

overwrite OR nooverwrite

overwrite

Files in the destination directory with the same name as sourceFileSpec will be overwritten.

nooverwrite

Files in the destination directory with the same name as sourceFileSpec will be preserved and the push will be flagged as incomplete to the destination directory. Files falling in this category will not be included in the return count.

copy OR move

copy

The file in the source directory will be left untouched after the push is completed.

move

The file in the source directory will be deleted after the push is completed.

maxwait= XX

Can be used to specify the maximum amount of time the Master will wait for a Remote to copy the file(s) to the intermediate transfer directory. The default is 240 seconds (i.e. "maxwait= 240s") in the absence of user specification for maxwait.

The syntax of "maxwait= xx" is very strict! There is no space between maxwait and the = sign. One space follows the = sign, followed by a number. An optional time unit designation character can be included following the number. Time units may be indicated using the following conventions: s=seconds, m=minutes, h=hours, d=days, w=weeks, mo=months (ex. "10s" = 10 seconds). In the absence of a time unit designation, seconds will be assumed.

pulledFiles() As String

After the function executes, this argument is populated with the full path name of each file that was successfully pulled from the remote nodes. The upper bound of the array is count-1.

Example

The example below demonstrates a typical distributed computing calculation where a FRED file (the one being run by the master) is pushed to the remote nodes, an embedded script within the FRED file is executed and FRED grid data (FGD) output files are pulled back to the master node for accumulation into a final result. Several helper functions are included to keep the code modular.

Sub Main

'This script demonstrates how to use the Distributed Computing component of FRED in order

'to send an analysis out to "remote" nodes on a Windows network, retrieve the results from

'each remote node and then recombine them on the "master" computer.

'Master document preparation

ARNDeleteAllNodes()

PrefsSetARNRetainCount( -1 )

ClearOutputWindow()

'Remove any FGD files in the current directory on the master node

Dim nDel As Long

nDel = KillFgd( GetDocDir() )

Print ""

Print "Removed " & nDel & " FGD files from the current directory."

'Load the connections() array from a CSV file

Dim nRemoteLoad As Long

Dim connections() As T_IPCINSTANCE

Dim configFile As String, configLocation As String

configLocation = GetDocDir() & "\"

configFile = "DCTestBed_Connections.csv"

nRemoteLoad = IPCLoadConfigFile( connections(), configLocation & configFile )

'Connect to the remote nodes

Dim nConnect As Long

nConnect = IPCConnect( connections() )

'Query the status of the remote nodes (info printed to output window).

Dim nStatus As Long

nStatus = IPCQueryStatus( connections(), "" )

'Push the FRED file to the remote nodes (file is the same as the file being run on master)

Dim nPush As Long, nDisconnect As Long

Dim fName As String

fName = name 'returns the name (with FRD extension) of the FRED document associated with this script

nPush = IPCPushFile( connections(), fName, "", "overwrite copy" )

If nPush < 1 Then

Print "No files were successfully pushed to the remotes. Terminating connections and stopping the master script."

nDisconnect = IPCTerminate( connections(), "sterilize", "" )

End

End If

'Load the model into the remote instances and execute an embedded script

Dim scriptName As String, nScript As Long, nLoad As Long

scriptName = "Remote Node Script"

nLoad = IPCLoadModel( connections(), fName, "1d" )

nScript = IPCExecEmbeddedScript( connections(), scriptName, "1d" )

'Have the remote instances push FGD files back to the master node. Embedded script was

'written to produce FGD files.

Dim nPull As Long

Dim pulledFiles() As String

nPull = IPCPullFile( connections(), "*.fgd", configLocation, "move", pulledFiles() )

'Print a report of the files that were pulled from the remotes

Dim curFile As String

Print ""

Print "The following files were successfully pulled from the remote nodes:"

For Each curFile In pulledFiles()

Print Chr(9) & curFile

'Terminate the connection to remotes.

nDisconnect = IPCTerminate( connections(), "sterilize", "" )

'Load FGD files from remotes into master FRED document and sum them together

'to form a final result.

Dim nFgd As Long, finalArn As Long

If nPull > 0 Then

nFgd = LoadFgd( pulledFiles() )

Print ""

Print "Loaded " & nFgd & " FGD files into ARNs."

finalArn = CompositeAllARN( )

Print "Final result stored in ARN " & finalArn

Else

Print "No files were pulled from remote nodes. No final result generated."

End If

Print ""

Print "Distributed calculation completed."

End Sub

Function LoadFgd( ByVal in_files() As String ) As Long

'This helper function loops over an array of strings that

'specify the full file path to a set of FGD files and then

'loads them into the FRED document as ARN.

'INPUTS:

' in_files() = string defining the directory to be searched

'OUTPUTS:

' returns the number of ARN that were created

Dim cFile As String, splitStrs() As String

Dim nF As Long

nF = 0

For Each cFile In in_files()

splitStrs = Split(cFile, "\") 'split the full file path name by "\"

splitStrs = Split( splitStrs(UBound(splitStrs)), ".") 'split the file name by "." to remove the file extension

ARNCreateFromFile( cFile, splitStrs(0) )

nF += 1

Return nF

End Function

Function CompositeAllARN( ) As Long

'This helper function loops over the Analysis Results folder

'and summs all of the ARN into a composite result.

'INPUTS:

' None

'OUTPUTS:

' Returns the node number of the composite ARN

'What node number is the last ARN on the tree at?

Dim nArn As Long

nArn = ARNGetMaxNodeNum()

'Make a copy of the zeroth ARN. This will become the

'final composite result node.

Dim compArn As Long

compArn = ARNCreateCopy( 0, "Composite Result" )

'Start looping over the other ARN starting at index 1

Dim curArn As Long

For curArn = 1 To nArn

ARNLinearCombination( 1, compArn, 1, curArn )

Return compArn

End Function

Function KillFgd( ByVal in_dir As String ) As Long

'This helper function scans a directory for FGD files

'and deletes them

'INPUTS:

' in_dir = string defining the directory to be searched

'OUTPUTS:

' returns the number of FGD files that were deleted

'Change to the search directory

ChDir( in_dir )

Dim nF As Long

Dim cFile As String

nF = 0

cFile = Dir$("*.fgd")

While cFile <> ""

Kill( cFile )

nF += 1

cFile = Dir$()

Wend

Return nF

End Function

The example code above produces the following content in the FRED output window:

Removed 0 FGD files from the current directory.

[MASTER]IPCLoadConfigFile: 3 of 3 connections read from 'C:\temp\DCTestBed_Connections.csv'

index active speed host status

1 true 100 computer1 ---

2 true 100 computer2 ---

3 true 100 computer3 ---

[MASTER]IPCConnect: 3 of 3 Remote FRED instances connected and running

index host result message

1 computer1 connected ---

2 computer2 connected ---

3 computer3 connected ---

[MASTER]IPCQueryStatus: 3 connections queried

index host message

1 computer1 idle

2 computer2 idle

3 computer3 idle

[MASTER]IPCPushFile: 3 files pushed from 'dctestbed.frd' to ''

index host src/xfer/dest message

1 computer1 1/1/1 finished

2 computer2 1/1/1 finished

3 computer3 1/1/1 finished

[MASTER]IPCLoadModel: 3 of 3 queued for loading file 'DCTestBed.frd'

[MASTER]IPCExecEmbeddedScript: 3 of 3 queued for execution 'Remote Node Script'

(static load balancing)

[MASTER]IPCPullFile: 3 files pulled from '*.fgd' to 'C:\temp\'

index host src/xfer/dest message

1 computer1 1/1/1 finished

2 computer2 1/1/1 finished

3 computer3 1/1/1 finished

The following files were successfully pulled from the remote nodes:

C:\temp\Color Image Result_computer1_lh_20160117135803_00000004.fgd

C:\temp\Color Image Result_computer2_lh_20160117135803_00000009.fgd

C:\temp\Color Image Result_computer3_lh_20160117135804_00000001.fgd

[MASTER]IPCTerminate: 3 of 3 Remote FRED instance terminations initiated

index host message

1 computer1 terminate - initiated

2 computer2 terminate - initiated

3 computer3 terminate - initiated

Loaded 3 FGD files into ARNs.

Final result stored in ARN 3

Distributed calculation completed.