Ziggy release 0.10.0 (2025-10-29)

Author: wohler

CITATION.cff

@@ -1,8 +1,8 @@
 cff-version: 1.2.0
 message: "If you use this software in your research, please cite it using these metadata."
 title: Ziggy
-version: v0.9.0
-date-released: "2025-04-10"
+version: v0.10.0
+date-released: "2025-10-29"
 abstract: "Ziggy, a portable, scalable infrastructure for science data processing pipelines, is the child of the Transiting Exoplanet Survey Satellite (TESS) pipeline and the grandchild of the Kepler Pipeline."
 authors:
 - family-names: Tenenbaum

RELEASE-NOTES.md

@@ -4,6 +4,54 @@
 
 These are the release notes for Ziggy. In the change log below, we'll refer to our internal Jira key for our benefit. If the item is associated with a resolved GitHub issue or pull request, we'll add a link to that. Changes that are incompatible with previous versions are marked below. While the major version is 0, we will be making breaking changes when bumping the minor number. However, once we hit version 1.0.0, incompatible changes will only be made when bumping the major number.
 
+# v0.10.0: Almost ready for 1.0.0
+
+This version is being used to run the Transiting Exoplanet Survey Satellite (TESS) data analysis pipeline. That effort is currently in dry run in preparation for formal V&V.
+
+The content of this release contains bug fixes and other improvements to get us to this point.
+
+## New Features
+
+1. Relax stringent pipeline configuration naming conventions (ZIGGY-258)
+1. Add java heap size control for before / after algorithm executor (ZIGGY-559)
+1. Adjust task request priorities (ZIGGY-560)
+1. Pipeline task priority in task request handlers is incorrect (ZIGGY-566)
+1. Add final subtask check to TaskMonitor (ZIGGY-568)
+1. Add action thread to ZiggyMessenger (ZIGGY-572)
+1. Remove node numbers from Edit pipeline dialog (ZIGGY-576)
+1. Improve performance of marshaling for new-data processing (ZIGGY-577)
+1. Move Supervisor PID file (ZIGGY-583)
+1. Allow 1-D HDF5 arrays to map to N-D Java fields (ZIGGY-584)
+1. Improvements to task restart API (ZIGGY-585)
+1. Move XmlSchemaExporter classes to command line (ZIGGY-589)
+1. Add index to pipeline instance in PipelineTask (ZIGGY-591)
+1. Delete ProtectedEntityInterceptor class (ZIGGY-596)
+1. Remove sandy bridge option for HECC (ZIGGY-601)
+1. Log "Subtask LOCKED" message only once (ZIGGY-607)
+
+## Bug Fixes
+
+1. The ziggy command does not perform property expansion (ZIGGY-235)
+1. Ziggy query to find consumed files runs too slowly (ZIGGY-548)
+1. Restart fails after marshaling failure (ZIGGY-558)
+1. Failures in remote execution management (ZIGGY-561)
+1. Untangle algorithm and worker heap size usage (ZIGGY-564)
+1. Pipeline step names cannot contain whitespace (ZIGGY-567)
+1. Task Information and Remote Execution commands throw exceptions (ZIGGY-570)
+1. ZiggyFileUtils error on nonexistent directory (ZIGGY-573)
+1. Resubmit tasks is unreliable (ZIGGY-574)
+1. Intermittent failures in ZiggyDatabaseRule (ZIGGY-575)
+1. Worker status panel displays wrong resources (ZIGGY-579)
+1. Resume current step doesn't update console (ZIGGY-580)
+1. Timeout hit viewing logs and log listings (ZIGGY-590)
+1. Edit pipelines dialog gives ConcurrentModificationException (ZIGGY-593)
+1. Ziggy's HDF5 installation refers to the outside directory (ZIGGY-597)
+1. Delete subtask lock files before resubmitting a task (ZIGGY-598)
+1. Double-check that all files are deleted before deleting a directory (ZIGGY-599)
+1. Create log directory for database on database start (ZIGGY-600)
+1. Speed up Instances Panel on console (ZIGGY-605)
+1. Supervisor crashes when orphaned remote jobs are present (ZIGGY-606)
+
 # v0.9.0: Getting ready for 1.0.0
 
 Since this was our last chance to rename everything, we did so. The architecture has diverged over the decades, but we've made the names and architecture a lot more consistent in this release. You'll notice some of the changes in the XML elements in your pipeline definition (the Ziggy definitions are now in etc/ziggy.d); the rest are internal. In particular, rather than using "module" in a gazillion contexts, we've teased out the differences in separate names. The algorithms are now called "steps." When those steps are incorporated into a "pipeline," they are called "nodes." We no longer use module except in the context of Python modules.

build.gradle

@@ -22,15 +22,6 @@ defaultTasks ':buildSrc:publish', 'publish', 'test'
 ext {
     // Location of third-party sources.
     outsideDir = "$rootDir/outside"
-
-    // The dependency group used for libraries from outside is called "outside".
-    // This variable contains the directory that contains the libraries from
-    // outside. Without the dependency group and the outside subdirectory,
-    // consumers of the published module will get "null" errors.
-    outsideGroupDir = "$buildDir/libs/outside"
-
-    // Name of the outside group, used in publishing artifacts.
-    outsideGroup = "outside"
 }
 
 repositories {
@@ -78,8 +69,9 @@ dependencies {
     implementation 'commons-beanutils:commons-beanutils:1.10.+'
 
     // Libraries built outside.
-    implementation 'outside:jarhdf5:1.14.+'
-    implementation 'outside:wrapper:3.5.+'
+    // Add these to the copyLibs' exclude list; otherwise, the
+    // libraries will get zeroed out when the task runs.
+    implementation fileTree("$buildDir/libs") {include "jarhdf5-*.jar", "wrapper-*.jar"}
 
     // Needed to run unit tests and at runtime.
     implementation 'org.hsqldb:hsqldb:2.7.+'
@@ -174,27 +166,23 @@ tasks.withType(com.github.spotbugs.snom.SpotBugsTask) {
     }
 }
 
-// Outside build files should copy their jars to $outsideGroupDir and
-// then make copyOutsideLibs depend on the task that performs that copy.
-// This is used to publish the outside jars.
+// Outside build files should install their jars in $buildDir/libs and
+// then make copyOutsideLibs depend on the task that installs the jars.
 task copyOutsideLibs
 compileJava.dependsOn copyOutsideLibs
 
-
 // Apply Ziggy Gradle script plugins.
 
-// A couple of the other scripts depend on integrationTest.
-apply from: "script-plugins/test.gradle"
-
 // Most scripts in alphabetical order.
+// The test.gradle file is out of order since a couple of the other
+// scripts depend on integrationTest.
 apply from: "script-plugins/copy.gradle"
+apply from: "script-plugins/test.gradle"
 apply from: "script-plugins/database-schemas.gradle"
 apply from: "script-plugins/eclipse.gradle"
 apply from: "script-plugins/hdf5.gradle"
 apply from: "script-plugins/misc.gradle"
+apply from: "script-plugins/publish.gradle"
 apply from: "script-plugins/wrapper.gradle"
 apply from: "script-plugins/xml-schemas.gradle"
 apply from: "script-plugins/ziggy-libraries.gradle"
-
-// Depends on versions set in hdf5.gradle and wrapper.gradle and copyBuildSrc from copy.gradle.
-apply from: "script-plugins/publish.gradle"

doc/legal/NASA-Corporate-CLA.pdf

@@ -36,6 +36,18 @@ As you can see, there are three attributes to a `step` element.
 
 This is pretty much what it sounds like. This is the unique name that you will use in your pipelines to define the order in which pipeline steps are executed. 
 
+###### Can I Include Whitespace in the Step Name?
+
+Short answer: you can, but you probably shouldn't. 
+
+Longer answer:
+
+At runtime, the step name is used to create names for log files and for the task directory. Neither of these use-cases can tolerate whitespace because they interact with file systems that can take a dim view of whitespace in a file name. In order to match the round peg of a step name with whitespace into the square hole of the file system's expectations, all whitespace in the step name is replaced with underscore characters when creating the log files and task directories. Hence, your step name of "my step" will become "my_step" in the context of log files and task directories. 
+
+This is already annoying enough, but it gets worse if you want to write some sort of script that loops over log files or over task directories to do post-processing on your pipeline results. At that point you'd need to remember to use the name with underscores rather than the original name of the pipeline step.
+
+Anyway, don't use whitespace in a step name if you have any way to avoid it. 
+
 ##### `file` Attribute
 
 This is the file that Ziggy will use to execute the pipeline step. This is actually an optional attribute. If you don't specify a `file`, Ziggy will look for an executable file that has the step's `name` attribute as the file name. 

doc/legal/NASA-Individual-CLA.pdf

@@ -33,6 +33,7 @@ generate-manifest       gov.nasa.ziggy.data.management.Manifest
 hsqlgui                 org.hsqldb.util.DatabaseManagerSwing
 import-pipelines        gov.nasa.ziggy.pipeline.definition.importer.PipelineDefinitionCli
 metrics                 gov.nasa.ziggy.metrics.report.MetricsCli
+property-value          gov.nasa.ziggy.services.config.ZiggyConfiguration
 perf-report             gov.nasa.ziggy.metrics.report.PerformanceReport
 update-pipelines        gov.nasa.ziggy.pipeline.definition.importer.PipelineDefinitionCli
 $ 

doc/user-manual/images/edit-pipeline.png

@@ -58,9 +58,14 @@
     -->
 
     <!--
-        To view SQL, uncomment the following, set the console's ThresholdFilter level to "DEBUG",
+        To view SQL, first uncomment the following logger.
+        To view output in Eclipse, set the console's ThresholdFilter level to "DEBUG",
         and add the following VM arguments to your run configuration in Eclipse.
         -Dlog4j2.configurationFile=etc/log4j2.xml -Dhibernate.show_sql=true -Dhibernate.format_sql=true -Dhibernate.use_sql_comments=true
+        To view output in the supervisor log, set the "gov.nasa.ziggy" logger above to debug and append
+        the hibernate property settings above to the ziggy.default.jvm.args property in ziggy.properties.
+        To view output in the worker log, set the "gov.nasa.ziggy" logger above to debug and uncomment
+        the hibernate property settings above in TaskRequestHandler.commandLine().
         <Logger name="org.hibernate.SQL" level="debug"/>
     -->
 

doc/user-manual/pipeline-algorithms.md

@@ -6,7 +6,6 @@
 
   <remoteEnvironment name="HECC" description="HECC" batchSystem="pbs" costUnit="SBU"
                      queueTimeMetricsClass="gov.nasa.ziggy.module.remote.NasQueueTimeMetrics">
-    <architecture name="san" description="Sandy Bridge" cores="16" ramGigabytes="32" cost="0.47"/>
     <architecture name="ivy" description="Ivy Bridge" cores="20" ramGigabytes="64" cost="0.66"/>
     <architecture name="has" description="Haswell" cores="24" ramGigabytes="128" cost="0.8"/>
     <architecture name="bro" description="Broadwell" cores="28" ramGigabytes="128" cost="1.0"/>