Model limitation via memory

[hwk114]

I have a 16GB Mac. I can run GPT-Oss 20B in LM Studio, but not in LlamaBarn. Is this expected?

[mdstaff]

It looks like the project has a strict isCompatible check for each model which takes 75% of your total system memory minus some overhead for running macOS. This ensures a better experience than running out of memory and using swap.

  /// Memory overhead reserved for macOS and other apps (in MB).
  /// This margin is also passed to llama-server via --fit-target so both
  /// LlamaBarn's predictions and llama-server's runtime checks use the same value.
  static let memOverheadMb: Double = 2048

See selectModels here which filters by isCompatible
https://github.com/ggml-org/LlamaBarn/blob/main/LlamaBarn/Catalog/ModelFamily.swift#L39-L57

See memoryRequirementSummary for more info here:
https://github.com/ggml-org/LlamaBarn/blob/main/LlamaBarn/Catalog/CatalogEntry%2BCompatibility.swift#L131-L152

[hwk114]

I was wondering if we could provide users with more memory and GPU configuration options, this would greatly help users with lower-spec machines run larger models properly.

[erusev]

provide users with more memory and GPU configuration options

We're working on it. The idea is to have defaults that make sure that models run well, but we should also let people override these defaults.