Syncing streams

[jchodera]

Is there another way to grab data from streams without using stream.sync(folder)?

For example, if I just want to pull down part of a stream in order to strip out waters and append to a trajectory where I've already done this to the first nframes frames, how can I do that without transferring the whole trajectory?

[jchodera]

Actually, it looks like trajectories are fragmented into numerous files, and that stream.sync(folder) only looks at filenames to tell if it should sync a file or not. (This may actually be problematic if the process dies partway through writing a trajectory to disk.)

That means that if I overwrite these files with, say, another version of the trajectories stripped of solvent, stream.sync(folder) will happily ignore these files and only sync new ones.

A couple of additional questions:

• Is there a way to get a list of files (via the Python API) that (a) are available, and (b) those that would be synced?
• Is there a way to sync tags? Or retrieve tags in some way?

[proteneer]

Correct, the fragments can be obtained by stream.partitions. And yes, sync
will ignore existing fragments.

list of files is currently not available, mainly because doing a recursive
os.listdir() turns out to be an insanely expensive operation (i've done
lots of benchmarking on this).

tags can be retrieved manually, and in fact, any file can be retrieved
manually via the stream.download method provided you know its name, I think
you need to do stream.download('tags/your_tag')

On Wed, Sep 3, 2014 at 8:58 PM, John Chodera notifications@github.com
wrote:

Actually, it looks like trajectories are fragmented into numerous files,
and that stream.sync(folder) only looks at filenames to tell if it should
sync a file or not. (This may actually be problematic if the process dies
partway through writing a trajectory to disk.)

That means that if I overwrite these files with, say, another version of
the trajectories stripped of solvent, stream.sync(folder) will happily
ignore these files and only sync new ones.

A couple of additional questions:

• Is there a way to get a list of files (via the Python API) that (a)
  are available, and (b) those that would be synced?
• Is there a way to sync tags? Or retrieve tags in some way?

—
Reply to this email directly or view it on GitHub
#56 (comment).

Yutong Zhao
Stanford University

www.proteneer.com | simbios.stanford.edu

[jchodera]

Thanks!

Is there also a way to get a list of tags? Is this just stream.tags?

[proteneer]

Tags are assumed to be something the project manager knows, since he's the
one that set it up.

On Wed, Sep 3, 2014 at 9:24 PM, John Chodera notifications@github.com
wrote:

Thanks!

Is there also a way to get a list of tags? Is this just stream.tags?

—
Reply to this email directly or view it on GitHub
#56 (comment).

Yutong Zhao
www.proteneer.com

[jchodera]

There is also a stream.download(), but there's no way to get a list of files we could actually download. Is there a simple way around this?

[jchodera]

@proteneer suggests we could construct a list from using partitions. Perhaps we could add a method to construct this list?