[Scip] Question about reducing memory usage of SCIP optimizer

[Abde Ali Kagalwalla]

Hi Gerald,

In my code, I did not create any explicit structures to store the variable
data. Every time I want to add a new column, I call a function which does
the following:
*********************************************************************************************************************
SCIP_VAR* var;
SCIP_CALL_ABORT( SCIPcreateVarBasic(scip, &var, varName, 0.0, 1.0, 1.0,
SCIP_VARTYPE_BINARY) );
SCIP_CALL_ABORT(SCIPvarSetRemovable(var, true));
SCIP_CALL_ABORT(SCIPvarSetInitial(var, true));
SCIPvarMarkDeletable(var);
SCIP_CALL_ABORT(SCIPaddPricedVar(scip, var, 1.0)); //Score is 1, not clear
how it helps??
//Add to relevant constraints
for (int r = 0; r < rows; r++){
        for (int c = 0; c < cols; c++){
                if (mskFrcInst->getConstrtPtr(r, c) != NULL && intensity(r,
c) > 0.00001){
                         SCIP_CALL_ABORT(SCIPaddCoefLinear(scip,
CDconstraints(r, c), var, intensity(r, c)));
                }
        }
}
SCIP_CALL_ABORT( SCIPreleaseVar(scip, &var) );
return SCIP_OKAY;
********************************************************************************************************
So, I don't think I am storing the variables anywhere explicitly. I do
store the matrix of linear constraints that I have, but the number of these
constraints does not change with pricing rounds. Do you think that this
manner of adding variables will cause memory to not be freed when variables
are deleted ?

Based on your previous email, I updated my code to setup variable deletion
and I have the following settings for the various parameters now:
**********************************************************************************************************

SCIP_CALL_ABORT( SCIPsetBoolParam(scip, "pricing/delvars", TRUE));
SCIP_CALL_ABORT( SCIPsetBoolParam(scip, "pricing/delvarsroot", TRUE));
SCIP_CALL_ABORT(SCIPsetIntParam(scip, "lp/colagelimit", 5)); //Reduced to
half the default
SCIP_CALL_ABORT(SCIPsetBoolParam(scip, "lp/cleanupcols", TRUE));
SCIP_CALL_ABORT(SCIPsetBoolParam(scip, "lp/cleanupcolsroot", TRUE));
*********************************************************************************************************

With this, I think the variable deletion is working, I think. The pricer
inserts a total of around 264k columns, but at the point of termination
there are 20k variables. I am sending a separate email to you with the full
log file.
Is there any other impact of turning on variable deletion ? Can it slow
down the convergence of the upper/lower bounds ? Also, for setting
lp/colagelimit, if I were to make it even lower could it hurt the optimizer
in some way ?

About the issue that LP relaxation is consuming most of the memory, I am
not certain that it is the case. The way I checked was that when the memory
usage was very high according to "top", I looked at the log file and the
last line of the log file was something like this:
*******************************************************************************
Iteration log . . .
Iteration:     1    Objective     =             4.700949
Reinitializing primal norms . . .
******************************************************************************
So this seemed like the LP relaxation step.

Again, thank you very much for your help, Gerald. I really appreciate it.
And apologies for the long email with so many details.

Regards,

Abde Ali



On Thu, Jul 24, 2014 at 1:41 AM, Gerald Gamrath <gamrath at zib.de> wrote:

> Dear Abde Ali,
>
>
>  Thank you very much for your response. I let SCIP delete variables by
>> setting the two flags (pricing/delvars and pricing/delvarsroot) and marking
>> each variable that I add as deletable. But it does not seem to help reduce
>> the memory usage.
>>
> does your pricer release the variables after creating and adding them to
> the problem? If you don't do this, but keep an array of all created
> variables in your structure, you need to catch the vardeleted event and
> remove the variables from your structures as well. Only if you do this and
> release the variable, the memory of the variable will be freed. You can
> have a look at the Coloring example to see how it is done there.
>
> You can increase the number of variables which are deleted by decreasing
> the colagelimit and setting cleanupcols(root) to TRUE as described in the
> FAQ.
>
>
>  I usually run my code with a 24 hour limit. For one of the test-cases, it
>> started running out of memory after around 10 hours.
>> In this duration, the number of variables inserted by the pricer was
>> around 10^6. I am not sure if any of them got deleted. On my system, the
>> optimization job was killed by the OS when memory usage reached around 32G.
>>
> Using 32GB for a million variables is quite a lot, do you have some
> variable data or other structures where you store data corresponding to the
> individual variables? Perhaps you can save some space there?
>
>
>  In order to terminate more gracefully and atleast report the current
>> results, I added a small feature at the end of each pricing round: I
>> computed the current memory usage using some system calls and if it
>> exceeded 25G, I set the SCIP solver time limit to the current clock time. I
>> have attached a log file with all the statistics reported by SCIP for the
>> optimization.
>>
> In the log file you attached, SCIP prints that the transformed problem has
> 37675 variables. How many did your pricer generate? Can you send me the
> complete log with SCIPs output during the optimization (best directly to me
> in order to not flood the mailing list)? I assume you created a lot more
> variables? Then the deletion from the problem seems to work, but I guess
> the memory is not freed because you did not release the variable.
>
>
>  Another thing I noticed was that the memory usage typically peaks during
>> the LP relaxation solving step. Is this typical ?
>>
> When the first LP is solved, this is typical, because the LP is copied to
> the LP solver. Later on, the LP solving should not increase the memory
> usage so much. Are you sure that it is the LP solving and not the calls of
> your pricer during the pricing loop?
>
> Best,
> Gerald
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listserv.zib.de/pipermail/scip/attachments/20140725/1e52797a/attachment.html>