exec() currently allows me to launch a process synchronously (Mode = 1) or asynchronously (Mode = 0). But if I do it asynchronously, I cannot track the result.
Consider this scenario:
Suppose I want to run a batch of 100 zorro scripts, but I only want to run four or eight at a time (due to my processor having four cores / eight logical threads, and for some reason, the scripts are single-threaded only). In this case, I would ideally like to launch a master zorro script: * First, it launches four zorros. * Next, it tracks to see which of the processes finished, maybe once a second. * Upon one process completion, it launches the next zorro. * Continue until all 100 scripts are completed.
For this to work, I would need some sort of handle to track each process, such as to detect its status and return code (if applicable).
Can exec() be enhanced to do this?
In the meantime, I suppose I do have two feasible workarounds: * Run four synchronous batch files simultaneously, each launching zorro over and over again. * Use the WinAPI directly to create and track process handles.
Anyway, you can probably do something like this with power shell.
Below is something I created last year, perhaps it can serve as inspiration at least.
Here basically a parallell block will run many commands in parallell (duh!), but it won´t continue with the next parallell block of tasks until every process in the previos parallell block has finished.
Code:
#Add parameter for index (replace=)
#Filter out less trades for OOS, to much garbage currently
Workflow Zorro-Workflow
{
InlineScript {
if([System.IO.File]::Exists("Y:ZorroLogGenerated.csv")) {
Remove-Item Y:ZorroLogGenerated.csv
}
if([System.IO.File]::Exists("Y:ZorroLogGenSortedTest.csv")){
Remove-Item Y:ZorroLogGenSortedTest.csv
}
if([System.IO.File]::Exists("Y:ZorroLogGeneratedFactoryResult.csv")){
Move-Item -Path Y:ZorroLogGeneratedFactoryResult.csv -Destination Y:ZorroLogGeneratedFactoryResult_$(get-date -f yyyy-MM-dd_HHmm).csv
}
}
#Parameters to Zorro are in order: BarPeriod, StopMultiplier, TpMultiplier, LifeTime
parallel
{
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 5 MiningRunner | Out-Null
Write-Host "Finished first batch"
}
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 10 MiningRunner| Out-Null
Write-Host "Finished second batch"
}
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 15 MiningRunner| Out-Null
Write-Host "Finished third batch"
}
}
parallel
{
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 5 MiningRunner| Out-Null
Write-Host "Finished fourth batch"
}
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 10 MiningRunner| Out-Null
Write-Host "Finished fifth batch"
}
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 15 MiningRunner| Out-Null
Write-Host "Finished sixth batch"
}
}
parallel
{
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 5 MiningRunner| Out-Null
Write-Host "Finished seventh batch"
}
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 10 MiningRunner| Out-Null
Write-Host "Finished eighth batch"
}
InlineScript {
Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 15 MiningRunner| Out-Null
Write-Host "Finished ninth batch"
}
}
InlineScript {
Write-Host "Starting OOS testing"
Import-Csv Y:ZorroLogGenerated.csv -delimiter "`t" | sort ProfitFactor -Descending | Export-Csv -Path Y:ZorroLogGenSortedTest.csv -NoTypeInformation
$csv = Import-Csv Y:ZorroLogGenSortedTest.csv | where {$_.Total -notlike '-*'}
#Create result file and add headers
$header = "Asset`tBarPeriod`tStopMultiplier`tTakeProfitMultiplier`tLifeTime`tTrades`tTotal`tWinPerTrade`tProfitFactor`te-Ratio`tStrategy"
$header | Set-Content 'Y:ZorroLogGeneratedFactoryResult.csv'
$totalLines = $csv.Count
#TODO: Add random filename to be able to run OOS tests in parallell
#probably using #([System.IO.Path]::GetRandomFileName()).Split(�.�)[0]
foreach ($line in $csv)
{
$lineIndex = $csv.IndexOf($line)+1
$asset = $line.Asset
$asset = $asset.replace('/','')
Write-Host "Running OOS for system $lineIndex of $totalLines"
(Get-Content 'Y:ZorroStrategyGenFactoryTemplate.c') -replace 'replaceMe', $line.Strategy | Set-Content 'Y:ZorroStrategyGenFactoryTest.c'
Y:ZorroZorro.exe -run -a $line.Asset -d SHORT -i $line.BarPeriod -i $line.StopMultiplier -i $line.TakeProfitMultiplier -i $line.LifeTime GenFactoryTest | Out-Null
Move-Item -Path Y:ZorroLogGenFactoryTest_$asset.png -Destination Y:ZorroLogGenPlotsGenFactoryTest_$asset_$lineIndex.png
}
Write-Host "OOS testing completed, all done!"
}
}
Re: exec() enhancement: return a handle for tracking
[Re: Dalla]
#474703 10/30/1823:4510/30/1823:45
Not yet, but I anticipate some similar problems to arise soon. QuantLib can really take a while to emulate options data, and it's not designed to do multi-threading, so I should prefer multi-process, according to Luigi Ballabio.
Thanks for the script. I probably should learn some Powershell... cmd is awkward for this task. Python also looks like a pretty good fit.
I still think I would be writing less scripting code if exec() is upgraded as described.