feat: update solver settings for CBA

[measrainsey]

Closes:

• Update solver settings for CBA #623

Changes proposed in this Pull Request

PR to add option to use specific solver for CBA MSV, add gurobi-simplex solver option, and to move CBA-related loggers for easier debugging

Tasks

• add gurobi-simplex solver setting
• add option to specific specific solver settings for CBA MSV extraction
• use highs-simplex as default solver for MSV
• move CBA solvers (MSV, reference grid, and project grids) to results/[run]/cba/logs/

Workflow

Config files

• Added gurobi-simplex as a solver option (not used as default anywhere)
• Added option to use separate solver settings for CBA MSV extraction - this is helpful as we might want to use a commercial solver for CBA MSV but we can use HiGHS for the rest of the CBA solves (reference, projects, etc)
• Set highs-simplex as default solver for MSV extraction

Logs

• For easier tracking of solve times later (especially once we move our runs to the cluster), I decided to move the logs from the logs/ folder to a folder called results/logs/cba/, with separate subfolders for msv/, reference/, project/ - for the different kinds of solves the CBA does.

Gurobi simplex settings

For the gurobi-simplex solver option, the following settings are listed by default:

  "gurobi-simplex": {
      "threads": 32,
      "method": 1,  # dual simplex
      "NodeMethod": 1,  # if MIP
      "Seed": 123,
      "AggFill": 0,
      "PreDual": 0,
      "FeasibilityTol": 1.0e-05,
      "OptimalityTol": 1.0e-05,
      "ScaleFlag": 0,  # 0 to turn off scaling (faster)
  },

ScaleFlag = 0 was chosen based on a quick and small test that showed that it led to a slightly quicker solve than other ScaleFlag values.

• resolution: 1H
• snapshot start: 2019-01-01
• snapshot end: 2019-01-07

Solver	ScaleFlag	2030 MSV solve	2040 MSV solve	Total
gurobi-default		1.72 s	2.10 s	3.82 s
gurobi-simplex	default (-1)	1.79 s	1.67 s	3.46 s
gurobi-simplex	0	1.65 s	1.63 s	3.28 s
gurobi-simplex	1	1.72 s	1.68 s	3.40 s
gurobi-simplex	2	1.73 s	1.69 s	3.42 s

Open issues

Notes

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• [ ] Changed dependencies are added to pixi.toml (using pixi add <dependency-name>).
• Changes in configuration options are added in config/config.default.yaml.
• [ ] Changes in configuration options are documented in doc/configtables/*.csv.
• Changes in configuration options are added in config/test/*.yaml.
• The multiple weather/climate years test is passing locally (using pixi run -e open-tyndp tyndp-cyears-test).
• [ ] Open-TYNDP SPDX license header added to all touched files.
• [ ] For new data sources or versions, these instructions have been followed.
• [ ] New rules are documented in the appropriate doc/*.rst files.
• A release note doc/release_notes.rst is added.
• [ ] Major features are documented with up-to-date information in README and doc/index.rst.
• [ ] Module docstrings added to new Python scripts.

[coroa]

/build-installer

[github-actions]

Signed Windows installer built successfully. Download artifact.

[coroa]

@daniel-rdt Worked correctly. See above.

Sorry for adding noise to your PR, @measrainsey .

[daniel-rdt]

Thank you @measrainsey for this PR. The settings look good to me, I only have minor comments on the location of the logs and PR description

[daniel-rdt]

I think one could drop the deepcopy as we are not reading the original snakemake.params.solving again but this is more robust, so let's keep it. Memory overhead is also negligable.

[daniel-rdt]

Did you get to this scale attribute by testing? Probably good to document in the PR description what the value is based on

[measrainsey]

Thanks @daniel-rdt for the suggestion!

I did a quick test for the ScaleFlag for the CBA MSV solve, and it looks like ScaleFlag: 0 is a bit faster, so for now I changed it to that (perhaps with more rigorous testing, a different attribute could be better).

• resolution: 1H
• snapshot start: 2019-01-01
• snapshot end: 2019-01-07

Solver	ScaleFlag	2030 MSV solve	2040 MSV solve	Total
gurobi-default		1.72 s	2.10 s	3.82 s
gurobi-simplex	default (-1)	1.79 s	1.67 s	3.46 s
gurobi-simplex	0	1.65 s	1.63 s	3.28 s
gurobi-simplex	1	1.72 s	1.68 s	3.40 s
gurobi-simplex	2	1.73 s	1.69 s	3.42 s

(also added to PR description)

[measrainsey]

@daniel-rdt Thanks for your review! I accepted your suggestions about the logs locations (and updated the PR description to reflect the new location).

For the gurobi-simplex settings, I edited the value based on a quick test (and also updated the PR description to reflect this) - but feel free to leave any remaining comments/concerns

[daniel-rdt]

Thank you @measrainsey for the changes and updating the PR description so thoroughly! I have one more thought on the ScaleFlag question. We can also have a brief chat about it later? might be quicker :)

[daniel-rdt]

While your test have shown slight speed up with ScaleFlag 0, I am contemplating if our best option wouldn't be to let Gurobi decide automatically how to scale the model (i.e. ScaleFlag -1). It could be that the speed up that we are seeing with 0 here, is only because of the small size of the test model (1 week). The settings will be used for MSV on a whole year, will they not?