How to import tomograms reconstructed/processed outside of RELION/nextpyp?

[sam-mahdi]

Hello,

I know RELION projects can be imported into nextpyp, but is there a way to import to tomograms processed by other programs? I have tried to generate a star file in relions format and use the star import option:
e.g.

loop_
_rlnTomoName #1 
_rlnVoltage #2 
_rlnSphericalAberration #3 
_rlnAmplitudeContrast #4 
_rlnMicrographOriginalPixelSize #5 
_rlnTomoHand #6 
_rlnTomoTiltSeriesPixelSize #7 
_rlnTomoTomogramBinning #10 
_rlnTomoSizeX #11 
_rlnTomoSizeY #12 
_rlnTomoSizeZ #13 
_rlnTomoReconstructedTomogram #14 
tomo13 300 2.7 0.1 1.328 1.0 2.656 16 511 720 116 tomo13_rec.mrc 

but I got the error:

Traceback (most recent call last):

10 | File "/opt/pyp/bin/run/pyp", line 4779, in <module>
11 | new_imagelist, parameters = globalmeta.TomoStar2meta(tomo_star_file, refine_star_file, rln_path=rln_path, parameters=parameters)
12 | File "/opt/pyp/src/pyp/inout/metadata/pyp_metadata.py", line 2019, in TomoStar2meta
13 | assert (os.path.exists(refinestar)), f"{refinestar} does not exist"
14 | AssertionError: None does not exist

I did not input a refinement star file (since I am only trying to upload processed reconstructions. Not sure if the error is because this is not an actual RELION project directory (was reconstructed in warp).

[sam-mahdi]

Just wanted to add, it appears using this method the paths/directory setup must follow RELIONs setup, e.g.
file_path = os.path.join(Path(tilt_series_star_file).parents[1], "external", name, f"{name}.mrc")

[abartesaghi]

You can’t directly import tomograms generated by external programs into nextPYP, except by manually replacing the .rec files, which is generally not recommended. The preferred workflow is to import the tilt-series alignments, which you may already have, and then reconstruct the tomograms within nextPYP. If your tilt-series alignments did not require manual adjustment, re-aligning them in nextPYP may work just as well, since that process is automated and usually runs quickly.

To import the tilt-series alignments, you need to provide the path to a folder containing IMOD's .xf and .tlt files for each tilt series. If you want the reconstructed tomograms to match the original dimensions, be sure to use the appropriate thickness and binning settings during reconstruction.

The import function you are trying to use is intended for projects that already include particles and their corresponding alignments from sub-volume averaging. The error message indicates that the required STAR file from 3D refinement was not provided. The import function does assume that your files were produced by Relion (version 4 or 5).

[sam-mahdi]

How can I find the option for importing aligned tilt stacks? I only see the options to import raw, star, project, or sessions when trying to import data.

Additionally I'd like to reconstruct outside of nextpyp to test different pre-processing pipelines, reconstruction methods, denoising methods, etc., and then import them into nextpyp for downstream refinement (similar to WARP-RELION pipeline). I think it'd be very useful to be able to import other types of pipelines/data, this could be done similar to how RELION does it using the star files using the path indicated in the star file (in contrast to hard-coded paths like the "external" setup relion uses for alignments).

[abartesaghi]

Note that you don’t import aligned tilt series themselves, you import their alignments and then apply those to the raw tilt series.

There isn’t a dedicated option specifically for importing tilt-series alignments. Instead, import your data using the Tomography (from raw data) option. Then, in the Pre-processing block, open the Tilt-series alignment tab and choose import (from path) as the alignment method.

The hard-coded paths are included to support importing data saved by Relion, since that’s the format it uses. This feature isn’t intended as a general-purpose importer for arbitrary .star files (which can vary widely in structure); it only works with .star files produced by Relion versions 4 and 5. Additionally, these imports do not include just the tomograms; they also bring in associated metadata such as movie frame alignment, CTF estimation, particle picking, and particle refinement.

Since most downstream processing in nextPYP operates on 2D projection data, importing tomograms has limited use. However, if you still want to import precomputed tomograms, for example, for particle picking or other supported operations, you would have to make sure the tomogram geometry, dimensions, and binning all match exactly.

If you have a specific program or type of metadata you would like to import, we would be happy to explore adding support for it.

[sam-mahdi]

From my understanding, importing the alignments just means you took motion corrected data from nextpyp and ran alignment externally, then import the xf and tlt to generate nextpyp reconstructed tomograms?

My main intention is to be able to use alternative pre-processing methods and then apply them to the downstream processes of nextpyp. For example, WARP reconstruction has improved ctf estimation that has shown to improve reconstruction quality, or alternative reconstruction methods like Cryolithe, or alternative denoising methods like cryosamba, isonet2, deepdewedge, etc. I think it'd be much easier to have a way to import a project than add support for the various pre-processing methods out there. RELION also utilizes 2D data for downstream processing and requires similar associated metadata, however these can be done by generating the relevant star files to point to the path for the data (I currently utilize WARP and IMOD for pre-processing and test various reconstruction and denoising methods on particle picking and refinement via RELION by converting their metadata into RELION readable star files).

[abartesaghi]

Yes, that is the current workflow for importing tilt-series alignments. You can also export tilt-series alignments in the same IMOD format (*.xf and *.tlt files) if you want to use them externally, for example, to generate your own tomograms.

Regarding denoising, nextPYP already supports Topaz Denoise, cryoCARE, IsoNet, noise2map, and IsoNet2, although there are, of course, many other methods available.

While Relion can now use 2D data, it still does not use the frame data for each tilt, which requires tracking additional metadata from motion correction for every tilt.

That said, it should be possible to implement an import function that takes all the necessary metadata (including information about frame and tilt-series alignment), along with the locations of the raw data and tomograms, and converts everything into a nextPYP project. Once again, it would be up to the user to ensure the imported tomograms have the correct geometry and dimensions. The user would also need to provide the corresponding half-tomograms required for applying many of the recent denoising algorithms. We will look into this further.