Jupyter Snippet P4M MIPLIBtest
Jupyter Snippet P4M MIPLIBtest
Download and solve all MIPLIB instances
The aim is to create a nice data set for investigating with machine learning techniques. The complete set of instances and log files are stored in a shared data directory. The original data can be obtained from [miplib.zib.de]. Here we are going to only use the older http://miplib2010.zib.de/download/miplib2010-1.1.3-benchmark.tgz instances (to avoid some of the really large instances).
Obtaining the data files
These have been downloaded and stored with the script. We are going to compress each file separately using bzip2 (as this is more efficent than the default gzip and can still be read by cplex). Warning some of the files here are big. The compressed version is 94Mb!
cd /max-software/data/miplib2010 # or wherever this should be stored
wget http://miplib2010.zib.de/download/miplib2010-1.1.3-benchmark.tgz
tar -xvzf miplib2010-1.1.3-benchmark.tgz
ln -s miplib2010*/instances/miplib2010 instances
foreach f (instances/*.gz) { gunzip $f; bzip2 -9 $f:r}
Testing all instances
We are want to run all instances as an LP only using CPLEX with three different solvers: primal simplex, dual simplex & the barrier method. These should produce 3 sets of outputs of the form problem.method.log where “problem” and “method” are the base name of the MPS file and one of primal/dual/barrier.
Below is the test script (can be written to file to run as a separate script or run from here)
## setup
#!/usr/bin/env python
import os,sys
where="/max-software/data/miplib2010"
batchFile="/max-software/data/miplib2010/cplexLP/runall.sh"
toFile=True
# Run CPLEX on every instance (or create a batch file to this effect)
if toFile: bf = open(batchFile,"w")
for path,mps in [ (p,f) for p,_,files in os.walk(f"{where}/instances") for f in files if f.endswith(".mps.bz2")]:
base=mps.replace(".mps.bz2","")
for alg,algName in [ (1,"primal"), (2,"dual"), (4,"barrier")]:
cmd = f'/usr/local/bin/cplex -c "read {path}/{mps}" "change prob lp" "disp prob stats"' \
f' "set thr 1" "set lp {alg}" "set clock 1" "opt" '\
f'> {where}/cplexLP/{base}.{algName}.log'
if toFile:
print(cmd,file=bf)
else:
os.system(cmd)
print(f"completed {mps} with {algName}")
if toFile: bf.close()
!cd /max-software/data/miplib2010; zip -9 cplexLP.zip cplexLP/*.log|wc -l; ls -l cplexLP.zip
# the next command should work but may not because of network restrictions on maxima
!scp /max-software/data/miplib2010/cplexLP.zip users.monash.edu.au:~/WWW/Files/Other
updating: cplexLP/30n20b8.barrier.log (deflated 59%)
updating: cplexLP/30n20b8.dual.log (deflated 58%)
updating: cplexLP/30n20b8.primal.log (deflated 63%)
updating: cplexLP/acc-tight5.barrier.log (deflated 59%)
updating: cplexLP/acc-tight5.dual.log (deflated 75%)
updating: cplexLP/acc-tight5.primal.log (deflated 69%)
updating: cplexLP/aflow40b.barrier.log (deflated 57%)
updating: cplexLP/aflow40b.dual.log (deflated 60%)
updating: cplexLP/aflow40b.primal.log (deflated 56%)
updating: cplexLP/air04.barrier.log (deflated 59%)
updating: cplexLP/air04.dual.log (deflated 69%)
updating: cplexLP/air04.primal.log (deflated 76%)
updating: cplexLP/app1-2.barrier.log (deflated 59%)
updating: cplexLP/app1-2.dual.log (deflated 59%)
updating: cplexLP/app1-2.primal.log (deflated 59%)
updating: cplexLP/ash608gpia-3col.barrier.log (deflated 74%)
updating: cplexLP/ash608gpia-3col.dual.log (deflated 58%)
updating: cplexLP/ash608gpia-3col.primal.log (deflated 83%)
updating: cplexLP/bab5.barrier.log (deflated 62%)
updating: cplexLP/bab5.dual.log (deflated 68%)
updating: cplexLP/bab5.primal.log (deflated 68%)
updating: cplexLP/beasleyC3.barrier.log (deflated 59%)
updating: cplexLP/beasleyC3.dual.log (deflated 58%)
updating: cplexLP/beasleyC3.primal.log (deflated 57%)
updating: cplexLP/biella1.barrier.log (deflated 58%)
updating: cplexLP/biella1.dual.log (deflated 71%)
updating: cplexLP/biella1.primal.log (deflated 71%)
updating: cplexLP/bienst2.barrier.log (deflated 57%)
updating: cplexLP/bienst2.dual.log (deflated 56%)
updating: cplexLP/bienst2.primal.log (deflated 60%)
updating: cplexLP/binkar10_1.barrier.log (deflated 59%)
updating: cplexLP/binkar10_1.dual.log (deflated 57%)
updating: cplexLP/binkar10_1.primal.log (deflated 57%)
updating: cplexLP/bley_xl1.barrier.log (deflated 59%)
updating: cplexLP/bley_xl1.dual.log (deflated 87%)
updating: cplexLP/bley_xl1.primal.log (deflated 70%)
updating: cplexLP/bnatt350.barrier.log (deflated 59%)
updating: cplexLP/bnatt350.dual.log (deflated 61%)
updating: cplexLP/bnatt350.primal.log (deflated 60%)
updating: cplexLP/core2536-691.barrier.log (deflated 58%)
updating: cplexLP/core2536-691.dual.log (deflated 77%)
updating: cplexLP/core2536-691.primal.log (deflated 80%)
updating: cplexLP/cov1075.barrier.log (deflated 57%)
updating: cplexLP/cov1075.dual.log (deflated 73%)
updating: cplexLP/cov1075.primal.log (deflated 60%)
updating: cplexLP/csched010.barrier.log (deflated 59%)
updating: cplexLP/csched010.dual.log (deflated 69%)
updating: cplexLP/csched010.primal.log (deflated 68%)
updating: cplexLP/danoint.barrier.log (deflated 59%)
updating: cplexLP/danoint.dual.log (deflated 61%)
updating: cplexLP/danoint.primal.log (deflated 63%)
updating: cplexLP/dfn-gwin-UUM.barrier.log (deflated 57%)
updating: cplexLP/dfn-gwin-UUM.dual.log (deflated 55%)
updating: cplexLP/dfn-gwin-UUM.primal.log (deflated 56%)
updating: cplexLP/eil33-2.barrier.log (deflated 58%)
updating: cplexLP/eil33-2.dual.log (deflated 60%)
updating: cplexLP/eil33-2.primal.log (deflated 60%)
updating: cplexLP/eilB101.barrier.log (deflated 58%)
updating: cplexLP/eilB101.dual.log (deflated 59%)
updating: cplexLP/eilB101.primal.log (deflated 63%)
updating: cplexLP/enlight13.barrier.log (deflated 58%)
updating: cplexLP/enlight13.dual.log (deflated 58%)
updating: cplexLP/enlight13.primal.log (deflated 58%)
updating: cplexLP/enlight14.barrier.log (deflated 58%)
updating: cplexLP/enlight14.dual.log (deflated 58%)
updating: cplexLP/enlight14.primal.log (deflated 58%)
updating: cplexLP/ex9.barrier.log (deflated 69%)
updating: cplexLP/ex9.dual.log (deflated 91%)
updating: cplexLP/ex9.primal.log (deflated 76%)
updating: cplexLP/glass4.barrier.log (deflated 58%)
updating: cplexLP/glass4.dual.log (deflated 56%)
updating: cplexLP/glass4.primal.log (deflated 56%)
updating: cplexLP/gmu-35-40.barrier.log (deflated 58%)
updating: cplexLP/gmu-35-40.dual.log (deflated 59%)
updating: cplexLP/gmu-35-40.primal.log (deflated 56%)
updating: cplexLP/iis-100-0-cov.barrier.log (deflated 61%)
updating: cplexLP/iis-100-0-cov.dual.log (deflated 64%)
updating: cplexLP/iis-100-0-cov.primal.log (deflated 66%)
updating: cplexLP/iis-bupa-cov.barrier.log (deflated 59%)
updating: cplexLP/iis-bupa-cov.dual.log (deflated 68%)
updating: cplexLP/iis-bupa-cov.primal.log (deflated 62%)
updating: cplexLP/iis-pima-cov.barrier.log (deflated 60%)
updating: cplexLP/iis-pima-cov.dual.log (deflated 64%)
updating: cplexLP/iis-pima-cov.primal.log (deflated 63%)
updating: cplexLP/lectsched-4-obj.barrier.log (deflated 59%)
updating: cplexLP/lectsched-4-obj.dual.log (deflated 59%)
updating: cplexLP/lectsched-4-obj.primal.log (deflated 56%)
updating: cplexLP/m100n500k4r1.barrier.log (deflated 62%)
updating: cplexLP/m100n500k4r1.dual.log (deflated 62%)
updating: cplexLP/m100n500k4r1.primal.log (deflated 61%)
updating: cplexLP/macrophage.barrier.log (deflated 58%)
updating: cplexLP/macrophage.dual.log (deflated 60%)
updating: cplexLP/macrophage.primal.log (deflated 64%)
updating: cplexLP/map18.barrier.log (deflated 70%)
updating: cplexLP/map18.dual.log (deflated 71%)
updating: cplexLP/map18.primal.log (deflated 75%)
updating: cplexLP/map20.barrier.log (deflated 62%)
updating: cplexLP/map20.dual.log (deflated 71%)
updating: cplexLP/map20.primal.log (deflated 74%)
updating: cplexLP/mcsched.barrier.log (deflated 60%)
updating: cplexLP/mcsched.dual.log (deflated 68%)
updating: cplexLP/mcsched.primal.log (deflated 63%)
updating: cplexLP/mik-250-1-100-1.barrier.log (deflated 57%)
updating: cplexLP/mik-250-1-100-1.dual.log (deflated 55%)
updating: cplexLP/mik-250-1-100-1.primal.log (deflated 56%)
updating: cplexLP/mine-166-5.barrier.log (deflated 60%)
updating: cplexLP/mine-166-5.dual.log (deflated 62%)
updating: cplexLP/mine-166-5.primal.log (deflated 64%)
updating: cplexLP/mine-90-10.barrier.log (deflated 59%)
updating: cplexLP/mine-90-10.dual.log (deflated 62%)
updating: cplexLP/mine-90-10.primal.log (deflated 61%)
updating: cplexLP/msc98-ip.barrier.log (deflated 71%)
updating: cplexLP/msc98-ip.dual.log (deflated 72%)
updating: cplexLP/msc98-ip.primal.log (deflated 71%)
updating: cplexLP/mspp16.barrier.log (deflated 57%)
updating: cplexLP/mspp16.dual.log (deflated 54%)
updating: cplexLP/mspp16.primal.log (deflated 66%)
updating: cplexLP/mzzv11.barrier.log (deflated 61%)
updating: cplexLP/mzzv11.dual.log (deflated 83%)
updating: cplexLP/mzzv11.primal.log (deflated 62%)
updating: cplexLP/n3div36.barrier.log (deflated 60%)
updating: cplexLP/n3div36.dual.log (deflated 56%)
updating: cplexLP/n3div36.primal.log (deflated 56%)
updating: cplexLP/n3seq24.barrier.log (deflated 58%)
updating: cplexLP/n3seq24.dual.log (deflated 60%)
updating: cplexLP/n3seq24.primal.log (deflated 75%)
updating: cplexLP/n4-3.barrier.log (deflated 57%)
updating: cplexLP/n4-3.dual.log (deflated 60%)
updating: cplexLP/n4-3.primal.log (deflated 60%)
updating: cplexLP/neos-1109824.barrier.log (deflated 59%)
updating: cplexLP/neos-1109824.dual.log (deflated 56%)
updating: cplexLP/neos-1109824.primal.log (deflated 58%)
updating: cplexLP/neos-1337307.barrier.log (deflated 60%)
updating: cplexLP/neos-1337307.dual.log (deflated 66%)
updating: cplexLP/neos-1337307.primal.log (deflated 65%)
updating: cplexLP/neos-1396125.barrier.log (deflated 58%)
updating: cplexLP/neos-1396125.dual.log (deflated 65%)
updating: cplexLP/neos-1396125.primal.log (deflated 65%)
updating: cplexLP/neos13.barrier.log (deflated 56%)
updating: cplexLP/neos13.dual.log (deflated 55%)
updating: cplexLP/neos13.primal.log (deflated 58%)
updating: cplexLP/neos-1601936.barrier.log (deflated 60%)
updating: cplexLP/neos-1601936.dual.log (deflated 82%)
updating: cplexLP/neos-1601936.primal.log (deflated 73%)
updating: cplexLP/neos18.barrier.log (deflated 60%)
updating: cplexLP/neos18.dual.log (deflated 62%)
updating: cplexLP/neos18.primal.log (deflated 61%)
updating: cplexLP/neos-476283.barrier.log (deflated 57%)
updating: cplexLP/neos-476283.dual.log (deflated 62%)
updating: cplexLP/neos-476283.primal.log (deflated 64%)
updating: cplexLP/neos-686190.barrier.log (deflated 59%)
updating: cplexLP/neos-686190.dual.log (deflated 55%)
updating: cplexLP/neos-686190.primal.log (deflated 61%)
updating: cplexLP/neos-849702.barrier.log (deflated 65%)
updating: cplexLP/neos-849702.dual.log (deflated 74%)
updating: cplexLP/neos-849702.primal.log (deflated 65%)
updating: cplexLP/neos-916792.barrier.log (deflated 56%)
updating: cplexLP/neos-916792.dual.log (deflated 55%)
updating: cplexLP/neos-916792.primal.log (deflated 57%)
updating: cplexLP/neos-934278.barrier.log (deflated 61%)
updating: cplexLP/neos-934278.dual.log (deflated 83%)
updating: cplexLP/neos-934278.primal.log (deflated 81%)
updating: cplexLP/net12.barrier.log (deflated 58%)
updating: cplexLP/net12.dual.log (deflated 67%)
updating: cplexLP/net12.primal.log (deflated 67%)
updating: cplexLP/netdiversion.barrier.log (deflated 74%)
updating: cplexLP/netdiversion.dual.log (deflated 69%)
updating: cplexLP/netdiversion.primal.log (deflated 79%)
updating: cplexLP/newdano.barrier.log (deflated 57%)
updating: cplexLP/newdano.dual.log (deflated 56%)
updating: cplexLP/newdano.primal.log (deflated 60%)
updating: cplexLP/noswot.barrier.log (deflated 57%)
updating: cplexLP/noswot.dual.log (deflated 57%)
updating: cplexLP/noswot.primal.log (deflated 57%)
updating: cplexLP/ns1208400.barrier.log (deflated 58%)
updating: cplexLP/ns1208400.dual.log (deflated 76%)
updating: cplexLP/ns1208400.primal.log (deflated 67%)
updating: cplexLP/ns1688347.barrier.log (deflated 58%)
updating: cplexLP/ns1688347.dual.log (deflated 58%)
updating: cplexLP/ns1688347.primal.log (deflated 56%)
updating: cplexLP/ns1758913.barrier.log (deflated 62%)
updating: cplexLP/ns1758913.dual.log (deflated 57%)
updating: cplexLP/ns1758913.primal.log (deflated 65%)
updating: cplexLP/ns1766074.barrier.log (deflated 59%)
updating: cplexLP/ns1766074.dual.log (deflated 56%)
updating: cplexLP/ns1766074.primal.log (deflated 56%)
updating: cplexLP/ns1830653.barrier.log (deflated 58%)
updating: cplexLP/ns1830653.dual.log (deflated 65%)
updating: cplexLP/ns1830653.primal.log (deflated 62%)
updating: cplexLP/opm2-z7-s2.barrier.log (deflated 65%)
updating: cplexLP/opm2-z7-s2.dual.log (deflated 70%)
updating: cplexLP/opm2-z7-s2.primal.log (deflated 75%)
updating: cplexLP/pg5_34.barrier.log (deflated 57%)
updating: cplexLP/pg5_34.dual.log (deflated 58%)
updating: cplexLP/pg5_34.primal.log (deflated 58%)
updating: cplexLP/pigeon-10.barrier.log (deflated 59%)
updating: cplexLP/pigeon-10.dual.log (deflated 58%)
updating: cplexLP/pigeon-10.primal.log (deflated 57%)
updating: cplexLP/pw-myciel4.barrier.log (deflated 58%)
updating: cplexLP/pw-myciel4.dual.log (deflated 64%)
updating: cplexLP/pw-myciel4.primal.log (deflated 60%)
updating: cplexLP/qiu.barrier.log (deflated 59%)
updating: cplexLP/qiu.dual.log (deflated 59%)
updating: cplexLP/qiu.primal.log (deflated 62%)
updating: cplexLP/rail507.barrier.log (deflated 59%)
updating: cplexLP/rail507.dual.log (deflated 59%)
updating: cplexLP/rail507.primal.log (deflated 60%)
updating: cplexLP/ran16x16.barrier.log (deflated 58%)
updating: cplexLP/ran16x16.dual.log (deflated 56%)
updating: cplexLP/ran16x16.primal.log (deflated 57%)
updating: cplexLP/reblock67.barrier.log (deflated 59%)
updating: cplexLP/reblock67.dual.log (deflated 62%)
updating: cplexLP/reblock67.primal.log (deflated 62%)
updating: cplexLP/rmatr100-p10.barrier.log (deflated 60%)
updating: cplexLP/rmatr100-p10.dual.log (deflated 62%)
updating: cplexLP/rmatr100-p10.primal.log (deflated 68%)
updating: cplexLP/rmatr100-p5.barrier.log (deflated 60%)
updating: cplexLP/rmatr100-p5.dual.log (deflated 64%)
updating: cplexLP/rmatr100-p5.primal.log (deflated 67%)
updating: cplexLP/rmine6.barrier.log (deflated 59%)
updating: cplexLP/rmine6.dual.log (deflated 59%)
updating: cplexLP/rmine6.primal.log (deflated 64%)
updating: cplexLP/rocII-4-11.barrier.log (deflated 58%)
updating: cplexLP/rocII-4-11.dual.log (deflated 55%)
updating: cplexLP/rocII-4-11.primal.log (deflated 56%)
updating: cplexLP/rococoC10-001000.barrier.log (deflated 59%)
updating: cplexLP/rococoC10-001000.dual.log (deflated 64%)
updating: cplexLP/rococoC10-001000.primal.log (deflated 59%)
updating: cplexLP/roll3000.barrier.log (deflated 60%)
updating: cplexLP/roll3000.dual.log (deflated 59%)
updating: cplexLP/roll3000.primal.log (deflated 61%)
updating: cplexLP/satellites1-25.barrier.log (deflated 60%)
updating: cplexLP/satellites1-25.dual.log (deflated 58%)
updating: cplexLP/satellites1-25.primal.log (deflated 62%)
updating: cplexLP/sp98ic.barrier.log (deflated 59%)
updating: cplexLP/sp98ic.dual.log (deflated 56%)
updating: cplexLP/sp98ic.primal.log (deflated 58%)
updating: cplexLP/sp98ir.barrier.log (deflated 58%)
updating: cplexLP/sp98ir.dual.log (deflated 59%)
updating: cplexLP/sp98ir.primal.log (deflated 61%)
updating: cplexLP/tanglegram1.barrier.log (deflated 66%)
updating: cplexLP/tanglegram1.dual.log (deflated 57%)
updating: cplexLP/tanglegram1.primal.log (deflated 81%)
updating: cplexLP/tanglegram2.barrier.log (deflated 59%)
updating: cplexLP/tanglegram2.dual.log (deflated 58%)
updating: cplexLP/tanglegram2.primal.log (deflated 71%)
updating: cplexLP/timtab1.barrier.log (deflated 58%)
updating: cplexLP/timtab1.dual.log (deflated 55%)
updating: cplexLP/timtab1.primal.log (deflated 55%)
updating: cplexLP/triptim1.barrier.log (deflated 60%)
updating: cplexLP/triptim1.dual.log (deflated 79%)
updating: cplexLP/triptim1.primal.log (deflated 78%)
updating: cplexLP/unitcal_7.barrier.log (deflated 64%)
updating: cplexLP/unitcal_7.dual.log (deflated 61%)
updating: cplexLP/unitcal_7.primal.log (deflated 69%)
updating: cplexLP/vpphard.barrier.log (deflated 58%)
updating: cplexLP/vpphard.dual.log (deflated 72%)
updating: cplexLP/vpphard.primal.log (deflated 73%)
updating: cplexLP/zib54-UUE.barrier.log (deflated 57%)
updating: cplexLP/zib54-UUE.dual.log (deflated 58%)
updating: cplexLP/zib54-UUE.primal.log (deflated 57%)
ssh: connect to host users.monash.edu.au port 22: Network is unreachable
lost connection