Changing FATES version

[evalieungh]

Hi! I want to change some of the FATES code using my own FATES fork and think we need some updates in the documentation.

With my fork of the LSP on NREC, I tried this:

1. Follow https://noresmhub.github.io/noresm-land-sites-platform/documentation/#ssh-tunneling-with-example-for-nrec
2. Make and attach new volume for better storage capacity, see Storage on NREC / Any files OK to delete after case creation? #157
3. Move to the lsp clone on the new volume
4. Make and move to new lsp branch (FATES_INCLINE)
5. Run a test case (OK!)
6. Change FATES external to own fork/branch (within lsp resources/model/components/clm/Externals_CLM.cfg):

[fates]
local_path = src/fates
protocol = git
repo_url = https://github.com/evalieungh/fates.git
# tag = sci.1.58.1_api.24.1.0
branch = grazing
required = True

[externals_description]
schema_version = 1.0.0

7. Change FATES remote url in resources/model/components/clm/src/fates and fetch all branches (git remote set-url origin https://github.com/evalieungh/fates.git, git fetch)
8. run ./manage_externals/checkout_externals from resources/model/components/clm
9. Reattach screen (screen -r name), bring containers down and up again
10. Run a test case - *case creation (create and setup case) was OK, but the run (build and submit case) failed.

OK, so two things:

First, checkout_externals is not enough

After step 6., I tried running ./manage_externals/checkout_externals but got an error about CLM not being in a clean state. It looked like step 7 fixed that, and checkout_externals ran through without errors.
I'm not sure I've done this correctly yet, but once we figure it out we need to update this part of the technical documentation: https://noresmhub.github.io/noresm-land-sites-platform/documentation/#model-versions

Second, I suspect our model versions are incompatible with new FATES releases already

I got this error from the lsp:

noresm-land-sites-platform-tasks-1     | [2022-12-12 09:55:01,792: INFO/ForkPoolWorker-2] Finished ./case.build in 143.85831952095032 seconds
noresm-land-sites-platform-tasks-1     | [2022-12-12 09:55:02,011: ERROR/ForkPoolWorker-2] Task app.tasks.cases.run_case[de934781-785e-411e-bd25-801ec6bfa20f] raised unexpected: Exception('Error: Missing actual argument for argument ‘bc_in’ at (1)\n\nError: Missing actual argument for argument ‘bc_in’ at (1)\n\nError: Missing actual argument for argument ‘bc_in’ at (1)\n\nERROR: BUILD FAIL: clm.buildlib failed, cat /ctsm-api/resources/cases/7d8538526916e39ab25eb3b9ca9f59b7_grazing-tst-0/bld/lnd.bldlog.221212-095238')
noresm-land-sites-platform-tasks-1     | Traceback (most recent call last):
noresm-land-sites-platform-tasks-1     |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
noresm-land-sites-platform-tasks-1     |     R = retval = fun(*args, **kwargs)
noresm-land-sites-platform-tasks-1     |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
noresm-land-sites-platform-tasks-1     |     return self.run(*args, **kwargs)
noresm-land-sites-platform-tasks-1     |   File "/ctsm-api/app/tasks/cases.py", line 182, in run_case
noresm-land-sites-platform-tasks-1     |     run_cmd(case, ["./case.build"], case_path, schemas.CaseRunStatus.BUILT)
noresm-land-sites-platform-tasks-1     |   File "/ctsm-api/app/tasks/cases.py", line 60, in run_cmd
noresm-land-sites-platform-tasks-1     |     raise Exception(proc.stderr.decode("utf-8").strip())
noresm-land-sites-platform-tasks-1     | Exception: Error: Missing actual argument for argument ‘bc_in’ at (1)
noresm-land-sites-platform-tasks-1     |
noresm-land-sites-platform-tasks-1     | Error: Missing actual argument for argument ‘bc_in’ at (1)
noresm-land-sites-platform-tasks-1     |
noresm-land-sites-platform-tasks-1     | Error: Missing actual argument for argument ‘bc_in’ at (1)
noresm-land-sites-platform-tasks-1     |
noresm-land-sites-platform-tasks-1     | ERROR: BUILD FAIL: clm.buildlib failed, cat /ctsm-api/resources/cases/7d8538526916e39ab25eb3b9ca9f59b7_grazing-tst-0/bld/lnd.bldlog.221212-095238
noresm-land-sites-platform-api-1       | INFO:     172.18.0.1:46872 - "GET /api/v1/cases/7d8538526916e39ab25eb3b9ca9f59b7 HTTP/1.1" 200 OK

Inside cases//bld/lnd.bldlog.221212-104446, I find

f951: Warning: Nonexistent include directory ‘/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/bld/gnu/mpich/nodebug/nothreads/nuopc/finclude’ [-Wmissing-include-dirs]
/ctsm-api/resources/model/components/clm/src/utils/clmfates_interfaceMod.F90:1839:37:

 1839 |                 this%fates(nc)%sites)
      |                                     1
Error: Missing actual argument for argument ‘bc_in’ at (1)
/ctsm-api/resources/model/components/clm/src/utils/clmfates_interfaceMod.F90:1667:72:

 1667 |                                                    this%fates(nc)%sites)
      |                                                                        1
Error: Missing actual argument for argument ‘bc_in’ at (1)
/ctsm-api/resources/model/components/clm/src/utils/clmfates_interfaceMod.F90:1091:63:

 1091 |                                           this%fates(nc)%sites)
      |                                                               1
Error: Missing actual argument for argument ‘bc_in’ at (1)
make: *** [/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/Tools/Makefile:972: clmfates_interfaceMod.o] Error 1

ERROR: Command make complib -j 4 COMP_NAME=clm COMPLIB=/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/bld/gnu/mpich/nodebug/nothreads/nuopc/nuopc/esmf/lib/libclm.a -f /ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/Tools/Makefile CIME_MODEL=cesm  SMP=FALSE CASEROOT="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1" CASETOOLS="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/Tools" CIMEROOT="/ctsm-api/resources/model/cime" SRCROOT="/ctsm-api/resources/model" COMP_INTERFACE="nuopc" COMPILER="gnu" DEBUG="FALSE" EXEROOT="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/bld" RUNDIR="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/run" INCROOT="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/bld/lib/include" LIBROOT="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/bld/lib" MACH="docker" MPILIB="mpich" NINST_VALUE="c1a1l1r1" OS="LINUX" PIO_VERSION=2 SHAREDLIBROOT="/ctsm-api/resources/cases/8ca381ad46cf4a1f6558a2ef0a3bc54c_grazing-tst-0-1/bld" SMP_PRESENT="FALSE" USE_ESMF_LIB="TRUE" USE_MOAB="FALSE" COMP_LND="clm" USE_TRILINOS="FALSE" USE_ALBANY="FALSE" USE_PETSC="FALSE"  failed with rc=2
/bc_in

It looks like https://github.com/ESCOMP/CTSM/blob/master/src/utils/clmfates_interfaceMod.F90#L1843 might be missing in the CLM version we use, i.e. our CLM version and the new FATES version are incompatible? I guess extra 'bc_in' lines were added to do something in newer versions of FATES and CLM.

[evalieungh]

Updated the CLM version too (to tag ctsm5.1.dev115) and now a test case ran successfully!