<div dir="ltr"><div><div>Hi Gerald,<br><br></div>In my code, I did not create any explicit structures to store the variable data. Every time I want to add a new column, I call a function which does the following: <br>*********************************************************************************************************************<br>
SCIP_VAR* var;<br>SCIP_CALL_ABORT( SCIPcreateVarBasic(scip, &var, varName, 0.0, 1.0, 1.0, SCIP_VARTYPE_BINARY) );<br>SCIP_CALL_ABORT(SCIPvarSetRemovable(var, true));<br>SCIP_CALL_ABORT(SCIPvarSetInitial(var, true));<br>
SCIPvarMarkDeletable(var);<br>SCIP_CALL_ABORT(SCIPaddPricedVar(scip, var, 1.0)); //Score is 1, not clear how it helps??<br>//Add to relevant constraints<br>for (int r = 0; r < rows; r++){<br> for (int c = 0; c < cols; c++){<br>
if (mskFrcInst->getConstrtPtr(r, c) != NULL && intensity(r, c) > 0.00001){<br> SCIP_CALL_ABORT(SCIPaddCoefLinear(scip, CDconstraints(r, c), var, intensity(r, c)));<br>
}<br> }<br>}<br>SCIP_CALL_ABORT( SCIPreleaseVar(scip, &var) );<br></div>return SCIP_OKAY;<br><div>********************************************************************************************************<br>
</div><div>So, I don't think I am storing the variables anywhere explicitly. I do store the matrix of linear constraints that I have, but the number of these constraints does not change with pricing rounds. Do you think that this manner of adding variables will cause memory to not be freed when variables are deleted ? <br>
<br></div><div>Based on your previous email, I updated my code to setup variable deletion and I have the following settings for the various parameters now:<br>********************************************************************************************************** <br>
</div><div>SCIP_CALL_ABORT( SCIPsetBoolParam(scip, "pricing/delvars", TRUE));<br>SCIP_CALL_ABORT( SCIPsetBoolParam(scip, "pricing/delvarsroot", TRUE));<br>SCIP_CALL_ABORT(SCIPsetIntParam(scip, "lp/colagelimit", 5)); //Reduced to half the default<br>
SCIP_CALL_ABORT(SCIPsetBoolParam(scip, "lp/cleanupcols", TRUE));<br>SCIP_CALL_ABORT(SCIPsetBoolParam(scip, "lp/cleanupcolsroot", TRUE));<br>*********************************************************************************************************<br>
<br></div><div>With this, I think the variable deletion is working, I think. The pricer inserts a total of around 264k columns, but at the point of termination there are 20k variables. I am sending a separate email to you with the full log file. <br>
</div><div>Is there any other impact of turning on variable deletion ? Can it slow down the convergence of the upper/lower bounds ? Also, for setting lp/colagelimit, if I were to make it even lower could it hurt the optimizer in some way ?<br>
<br></div><div>About the issue that LP relaxation is consuming most of the memory, I am not certain that it is the case. The way I checked was that when the memory usage was very high according to "top", I looked at the log file and the last line of the log file was something like this: <br>
*******************************************************************************<br>Iteration log . . .<br>Iteration: 1 Objective = 4.700949<br>Reinitializing primal norms . . .<br>******************************************************************************<br>
</div><div>So this seemed like the LP relaxation step. <br><br></div><div>Again, thank you very much for your help, Gerald. I really appreciate it. And apologies for the long email with so many details. <br><br></div><div>
Regards,<br><br></div><div>Abde Ali <br></div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 24, 2014 at 1:41 AM, Gerald Gamrath <span dir="ltr"><<a href="mailto:gamrath@zib.de" target="_blank">gamrath@zib.de</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Abde Ali,<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thank you very much for your response. I let SCIP delete variables by setting the two flags (pricing/delvars and pricing/delvarsroot) and marking each variable that I add as deletable. But it does not seem to help reduce the memory usage.<br>
</blockquote></div>
does your pricer release the variables after creating and adding them to the problem? If you don't do this, but keep an array of all created variables in your structure, you need to catch the vardeleted event and remove the variables from your structures as well. Only if you do this and release the variable, the memory of the variable will be freed. You can have a look at the Coloring example to see how it is done there.<br>
<br>
You can increase the number of variables which are deleted by decreasing the colagelimit and setting cleanupcols(root) to TRUE as described in the FAQ.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I usually run my code with a 24 hour limit. For one of the test-cases, it started running out of memory after around 10 hours.<br>
In this duration, the number of variables inserted by the pricer was around 10^6. I am not sure if any of them got deleted. On my system, the optimization job was killed by the OS when memory usage reached around 32G.<br>
</blockquote></div>
Using 32GB for a million variables is quite a lot, do you have some variable data or other structures where you store data corresponding to the individual variables? Perhaps you can save some space there?<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
In order to terminate more gracefully and atleast report the current results, I added a small feature at the end of each pricing round: I computed the current memory usage using some system calls and if it exceeded 25G, I set the SCIP solver time limit to the current clock time. I have attached a log file with all the statistics reported by SCIP for the optimization.<br>
</blockquote></div>
In the log file you attached, SCIP prints that the transformed problem has 37675 variables. How many did your pricer generate? Can you send me the complete log with SCIPs output during the optimization (best directly to me in order to not flood the mailing list)? I assume you created a lot more variables? Then the deletion from the problem seems to work, but I guess the memory is not freed because you did not release the variable.<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Another thing I noticed was that the memory usage typically peaks during the LP relaxation solving step. Is this typical ?<br>
</blockquote></div>
When the first LP is solved, this is typical, because the LP is copied to the LP solver. Later on, the LP solving should not increase the memory usage so much. Are you sure that it is the LP solving and not the calls of your pricer during the pricing loop?<br>
<br>
Best,<br>
Gerald<br>
<br>
</blockquote></div><br></div>