IPCPushFile Function

Navigation: Scripting Reference Manual > Functions & Subroutines > IPCPushFile

IPCPushFile

Description

This function pushes (copies) the given file (or files) from sourceFileSpec to destinationDirectory on each remote node. That is, each individual file in sourceFileSpec is copied to the destinationDirectory of every remote node. The actual file copy occurs in two steps. In the first step, the master node copies the file(s) from the source to the intermediate file transfer directory specified in the remote node's connection. In the second step, a command is sent to the remote nodes to copy the file(s) from the intermediate directory to their destination directory. The master then waits until the remote nodes report that the files have been copied.

The text output of the IPCPushFile() command lists how many source files were found, how many were successfully copied to the intermediate transfer directory, and how many were successfully copied from the intermediate directory to the destination directory.

Syntax

count = IPCPushFile( connections(), sourceFileSpec, destinationDirectory, options )

Parameters

count (Long)

Returned number of files that were successfully pushed to the destination directories of the remote nodes.

connections() As T_IPCINSTANCE

An array of T_IPCINSTANCE structures, with each structure defining the connection information for a remote node.

sourceFileSpec As String

The name of the file that will be pushed to the remote nodes (may include wildcard characters, resulting in multiple files being pushed). The file(s) typically reside on the master node and are pushed to the remote nodes. If the full path name of the file is not specified, the current working directory will be searched for sourceFileSpec. The program's current working directory can be checked by issuing, Print CurDir(). The current working directory can be changed to that of the FRED document running the master script by issuing, ChDir( GetDocDir() ).

destinationDirectory As String

Full path to the directory where sourceFileSpec should be pushed as specified in the local directory structure of each remote computer. If an empty string is used (typical), the default working directory of each remote instance is used.

options As String

Space delimited string specifying options for file overwrite and move behavior, with the default empty string indicating "overwrite copy". Options are the following:

overwrite OR nooverwrite

overwrite

Files in the destination directory with the same name as sourceFileSpec will be overwritten.

nooverwrite

Files in the destination directory with the same name as sourceFileSpec will be preserved and the push will be flagged as incomplete to the destination directory. Files falling in this category will not be included in the return count.

copy OR move

copy

The file in the source directory will be left untouched after the push is completed.

move

The file in the source directory will be deleted after the push is completed.

maxwait= XX

Can be used to specify the maximum amount of time the Master will wait for a Remote to copy the file(s) from the intermediate transfer directory to the destination directory. The default is 240 seconds (i.e. "maxwait= 240s") in the absence of user specification for maxwait.

The syntax of "maxwait= xx" is very strict! There is no space between maxwait and the = sign. One space follows the = sign, followed by a number. An optional time unit designation character can be included following the number. Time units may be indicated using the following conventions: s=seconds, m=minutes, h=hours, d=days, w=weeks, mo=months (ex. "10s" = 10 seconds). In the absence of a time unit designation, seconds will be assumed.

Example

The example below demonstrates a typical distributed computing calculation where a FRED file (the one being run by the master) is pushed to the remote nodes, an embedded script within the FRED file is executed and FRED grid data (FGD) output files are pulled back to the master node for accumulation into a final result. Several helper functions are included to keep the code modular.

Sub Main

'This script demonstrates how to use the Distributed Computing component of FRED in order

'to send an analysis out to "remote" nodes on a Windows network, retrieve the results from

'each remote node and then recombine them on the "master" computer.

'Master document preparation

ARNDeleteAllNodes()

PrefsSetARNRetainCount( -1 )

ClearOutputWindow()

'Remove any FGD files in the current directory on the master node

Dim nDel As Long

nDel = KillFgd( GetDocDir() )

Print ""

Print "Removed " & nDel & " FGD files from the current directory."

'Load the connections() array from a CSV file

Dim nRemoteLoad As Long

Dim connections() As T_IPCINSTANCE

Dim configFile As String, configLocation As String

configLocation = GetDocDir() & "\"

configFile = "DCTestBed_Connections.csv"

nRemoteLoad = IPCLoadConfigFile( connections(), configLocation & configFile )

'Connect to the remote nodes

Dim nConnect As Long

nConnect = IPCConnect( connections() )

'Query the status of the remote nodes (info printed to output window).

Dim nStatus As Long

nStatus = IPCQueryStatus( connections(), "" )

'Push the FRED file to the remote nodes (file is the same as the file being run on master)

Dim nPush As Long, nDisconnect As Long

Dim fName As String

fName = name 'returns the name (with FRD extension) of the FRED document associated with this script

nPush = IPCPushFile( connections(), fName, "", "overwrite copy" )

If nPush < 1 Then

Print "No files were successfully pushed to the remotes. Terminating connections and stopping the master script."

nDisconnect = IPCTerminate( connections(), "sterilize", "" )

End

End If

'Load the model into the remote instances and execute an embedded script

Dim scriptName As String, nScript As Long, nLoad As Long

scriptName = "Remote Node Script"

nLoad = IPCLoadModel( connections(), fName, "1d" )

nScript = IPCExecEmbeddedScript( connections(), scriptName, "1d" )

'Have the remote instances push FGD files back to the master node. Embedded script was

'written to produce FGD files.

Dim nPull As Long

Dim pulledFiles() As String

nPull = IPCPullFile( connections(), "*.fgd", configLocation, "move", pulledFiles() )

'Print a report of the files that were pulled from the remotes

Dim curFile As String

Print ""

Print "The following files were successfully pulled from the remote nodes:"

For Each curFile In pulledFiles()

Print Chr(9) & curFile

'Terminate the connection to remotes.

nDisconnect = IPCTerminate( connections(), "sterilize", "" )

'Load FGD files from remotes into master FRED document and sum them together

'to form a final result.

Dim nFgd As Long, finalArn As Long

If nPull > 0 Then

nFgd = LoadFgd( pulledFiles() )

Print ""

Print "Loaded " & nFgd & " FGD files into ARNs."

finalArn = CompositeAllARN( )

Print "Final result stored in ARN " & finalArn

Else

Print "No files were pulled from remote nodes. No final result generated."

End If

Print ""

Print "Distributed calculation completed."

End Sub

Function LoadFgd( ByVal in_files() As String ) As Long

'This helper function loops over an array of strings that

'specify the full file path to a set of FGD files and then

'loads them into the FRED document as ARN.

'INPUTS:

' in_files() = string defining the directory to be searched

'OUTPUTS:

' returns the number of ARN that were created

Dim cFile As String, splitStrs() As String

Dim nF As Long

nF = 0

For Each cFile In in_files()

splitStrs = Split(cFile, "\") 'split the full file path name by "\"

splitStrs = Split( splitStrs(UBound(splitStrs)), ".") 'split the file name by "." to remove the file extension

ARNCreateFromFile( cFile, splitStrs(0) )

nF += 1

Return nF

End Function

Function CompositeAllARN( ) As Long

'This helper function loops over the Analysis Results folder

'and summs all of the ARN into a composite result.

'INPUTS:

' None

'OUTPUTS:

' Returns the node number of the composite ARN

'What node number is the last ARN on the tree at?

Dim nArn As Long

nArn = ARNGetMaxNodeNum()

'Make a copy of the zeroth ARN. This will become the

'final composite result node.

Dim compArn As Long

compArn = ARNCreateCopy( 0, "Composite Result" )

'Start looping over the other ARN starting at index 1

Dim curArn As Long

For curArn = 1 To nArn

ARNLinearCombination( 1, compArn, 1, curArn )

Return compArn

End Function

Function KillFgd( ByVal in_dir As String ) As Long

'This helper function scans a directory for FGD files

'and deletes them

'INPUTS:

' in_dir = string defining the directory to be searched

'OUTPUTS:

' returns the number of FGD files that were deleted

'Change to the search directory

ChDir( in_dir )

Dim nF As Long

Dim cFile As String

nF = 0

cFile = Dir$("*.fgd")

While cFile <> ""

Kill( cFile )

nF += 1

cFile = Dir$()

Wend

Return nF

End Function

The example code above produces the following content in the FRED output window:

Removed 0 FGD files from the current directory.

[MASTER]IPCLoadConfigFile: 3 of 3 connections read from 'C:\temp\DCTestBed_Connections.csv'

index active speed host status

1 true 100 computer1 ---

2 true 100 computer2 ---

3 true 100 computer3 ---

[MASTER]IPCConnect: 3 of 3 Remote FRED instances connected and running

index host result message

1 computer1 connected ---

2 computer2 connected ---

3 computer3 connected ---

[MASTER]IPCQueryStatus: 3 connections queried

index host message

1 computer1 idle

2 computer2 idle

3 computer3 idle

[MASTER]IPCPushFile: 3 files pushed from 'dctestbed.frd' to ''

index host src/xfer/dest message

1 computer1 1/1/1 finished

2 computer2 1/1/1 finished

3 computer3 1/1/1 finished

[MASTER]IPCLoadModel: 3 of 3 queued for loading file 'DCTestBed.frd'

[MASTER]IPCExecEmbeddedScript: 3 of 3 queued for execution 'Remote Node Script'

(static load balancing)

[MASTER]IPCPullFile: 3 files pulled from '*.fgd' to 'C:\temp\'

index host src/xfer/dest message

1 computer1 1/1/1 finished

2 computer2 1/1/1 finished

3 computer3 1/1/1 finished

The following files were successfully pulled from the remote nodes:

C:\temp\Color Image Result_computer1_lh_20160117135803_00000004.fgd

C:\temp\Color Image Result_computer2_lh_20160117135803_00000009.fgd

C:\temp\Color Image Result_computer3_lh_20160117135804_00000001.fgd

[MASTER]IPCTerminate: 3 of 3 Remote FRED instance terminations initiated

index host message

1 computer1 terminate - initiated

2 computer2 terminate - initiated

3 computer3 terminate - initiated

Loaded 3 FGD files into ARNs.

Final result stored in ARN 3

Distributed calculation completed.