[Scip] Question about reducing memory usage of SCIP optimizer

[Gerald Gamrath]

Dear Abde Ali,

> Thank you very much for your response. I let SCIP delete variables by 
> setting the two flags (pricing/delvars and pricing/delvarsroot) and 
> marking each variable that I add as deletable. But it does not seem to 
> help reduce the memory usage.
does your pricer release the variables after creating and adding them to 
the problem? If you don't do this, but keep an array of all created 
variables in your structure, you need to catch the vardeleted event and 
remove the variables from your structures as well. Only if you do this 
and release the variable, the memory of the variable will be freed. You 
can have a look at the Coloring example to see how it is done there.

You can increase the number of variables which are deleted by decreasing 
the colagelimit and setting cleanupcols(root) to TRUE as described in 
the FAQ.

> I usually run my code with a 24 hour limit. For one of the test-cases, 
> it started running out of memory after around 10 hours.
> In this duration, the number of variables inserted by the pricer was 
> around 10^6. I am not sure if any of them got deleted. On my system, 
> the optimization job was killed by the OS when memory usage reached 
> around 32G.
Using 32GB for a million variables is quite a lot, do you have some 
variable data or other structures where you store data corresponding to 
the individual variables? Perhaps you can save some space there?

> In order to terminate more gracefully and atleast report the current 
> results, I added a small feature at the end of each pricing round: I 
> computed the current memory usage using some system calls and if it 
> exceeded 25G, I set the SCIP solver time limit to the current clock 
> time. I have attached a log file with all the statistics reported by 
> SCIP for the optimization.
In the log file you attached, SCIP prints that the transformed problem 
has 37675 variables. How many did your pricer generate? Can you send me 
the complete log with SCIPs output during the optimization (best 
directly to me in order to not flood the mailing list)? I assume you 
created a lot more variables? Then the deletion from the problem seems 
to work, but I guess the memory is not freed because you did not release 
the variable.

> Another thing I noticed was that the memory usage typically peaks 
> during the LP relaxation solving step. Is this typical ?
When the first LP is solved, this is typical, because the LP is copied 
to the LP solver. Later on, the LP solving should not increase the 
memory usage so much. Are you sure that it is the LP solving and not the 
calls of your pricer during the pricing loop?

Best,
Gerald