Nellie
Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy
Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy¶
arXiv
Abstract: The analysis of dynamic organelles remains a formidable challenge, though key to understanding biological processes. We introduce Nellie, an automated and unbiased pipeline for segmentation, tracking, and feature extraction of diverse intracellular structures. Nellie adapts to image metadata, eliminating user input. Nellie’s preprocessing pipeline enhances structural contrast on multiple intracellular scales allowing for robust hierarchical segmentation of sub-organellar regions. Internal motion capture markers are generated and tracked via a radius-adaptive pattern matching scheme, and used as guides for sub-voxel flow interpolation. Nellie extracts a plethora of features at multiple hierarchical levels for deep and customizable analysis. Nellie features a Napari-based GUI that allows for code-free operation and visualization, while its modular open-source codebase invites customization by experienced users.
Nellie's pipeline and Napari plugin are both very much in early stages, therefore I highly encourage any and all feedback.
Example output intermediates¶
https://github.com/aelefebv/nellie/assets/26515909/1df8bf1b-7116-4d19-b5fb-9658f744675b
Installation (~ 1 minute)¶
Notes:
- It is recommended (but usually not required) to create a new environment for Nellie to avoid conflicts with other packages.
- May take several minutes to install.
- Choose one of the following methods, and only one!
- If you do not already have Python 3.9 or higher installed, download it via the python website.
https://github.com/user-attachments/assets/50b1cd4b-6df7-4f19-8db3-4dcc03388513
Option 1. Via Napari plugin manager:¶
If not already installed, install Napari: https://napari.org/stable/tutorials/fundamentals/installation
https://github.com/user-attachments/assets/0d44abe5-f575-4bd4-962a-2c102faf737c
- Open Napari
- Go to
Plugins > Install/Uninstall Plugins...
- Search for Nellie and click
Install
- Make sure Nellie is updated to the latest version.
- Restart Napari.
Option 2. Via PIP:¶
https://github.com/user-attachments/assets/b63df093-e3e1-49cb-925b-7efce36b9015
python3 -m pip install nellie
Option 2a for NVIDIA GPU acceleration, optional (Windows, Linux):¶
To use GPU acceleration via NVIDIA GPUs, you also need to install cupy:
pip install cupy-cudaXXx
- replace
cupy-cudaXXx
with the appropriate version for your CUDA version.- i.e.
cupy-cuda11x
for CUDA 11.x orcupy-cuda12x
for CUDA 12.x
- i.e.
- if you don't have CUDA installed, go here.
- Mac Metal GPU-acceleration coming... eventually.
Usage¶
The sample dataset shown below is in the repo if you want to play around without, and can be downloaded here.
General data preparation¶
- It is strongly recommended to have your data in a parsable format, such as .ome.tif, .nd2, or other raw data files from microscopes.
- Importing into ImageJ/FIJI and saving via BioFormats with the proper image dimensions should do the trick.
- If the metadata cannot be parsed, you will have to manually enter it.
- It is also recommended to crop your image as much as possible to reduce processing time and memory usage. But really, unless you have massive lightsheet data, it should be pretty fast (seconds to minutes on a typical modern desktop computer).
3D + Timeseries dataset¶
https://github.com/user-attachments/assets/531f76ee-f58e-4058-b5dc-4fdf09af3660
3D (no Timeseries) dataset¶
https://github.com/user-attachments/assets/30d55bfa-bade-4987-88f0-255bb36cb7e8
2D + Timeseries dataset¶
https://github.com/user-attachments/assets/d534c6e1-df31-4964-9c12-edff56228be3
Running Nellie's processing pipeline¶
- Start Napari (open a Terminal and type napari)
napari
- Go to
Plugins > Nellie (nellie)
then to theFile select
tab. - Click
Select File
ofSelect Folder
to select your image(s).- If the metadata boxes do not fill in automatically and turn red, this means Nellie did not detect that metadata portion from your image, and you must manually enter it or reformat your image and try again.
- The metadata slot will appear green if it is in the correct format.
- *Note, if you are batch processing, the metadata must be the same for all images if any of them are in an incorrect format (this will be fixed eventually). If they are different, but all pass validation, then it will process fine.
- You can preview your image via the
Open preview
button once the metadata is filled in to ensure it looks correct. - From this tab, you can also choose what time points and channel you want to analyze, if your file contains more than one slice in those dimensions.
- If the metadata boxes do not fill in automatically and turn red, this means Nellie did not detect that metadata portion from your image, and you must manually enter it or reformat your image and try again.
- Click the
Process
tab. - You can run the full pipeline with
Run Nellie
, or run individual steps below.- Steps can only be run once its previous step has been run.
- Likewise, visualizations in the
Visualization
tab can only be opened once its respective step has been run.
- All intermediate files and output csvs will be saved to
[image_directory]/nellie_output/
, which can be accessed via theOpen output directory
button.- A separate .csv is created for each level of the organellar hierarchy.
- Once features have been exported, Nellie will automatically detect this, and allow analysis via the
Analyze
tab.- Analysis at this point is optional, but can be helpful for visualizing, and selectively exporting data.
Using Nellie's visualization plugin¶
- Follow the previous processing steps, you only need to do this once per file as long as you don't move or delete the files.
- Open the
Visualization
tab - Select a visualization from the list.
Raw
: Visualize the raw data for the processed channel.Preprocessed
: Visualize the contrast-enhanced data.Segmentation
: Visualize the organelle and branch instance segmentation masks.Mocap Markers
: Visualize the mocap markers used for waypoints.Reassigned Labels
: Visualize the organelle and branch instance segmentation masks where voxels are reassigned based on the first timepoint.
- To visualize tracks, open and select one of the segmentation layers.
- To visualize all tracks of all organelles/branches, click the
Visualize all frame labels' tracks
button. - To visualize all tracks of a specific organelle/branch:
- Click on the layer, and use the eyedropper tool at the top to select an organelle/branch to track.
- Click the
Visualize selected label's tracks
.
Using Nellie's analysis plugin¶
- Follow the previous processing steps, you only need to do this once per file as long as you don't move or delete the files.
- Open the
Analyze
tab, select the hierarchy level you want to visualize from the dropdown. - Select the level-specific feature you want to visualize from the new dropdown.
- A histogram of all the data will be displayed.
- This histogram can be directly exported via the
Save graph
button. A .png will be saved to[image_directory]/nellie_output/graphs/
with the current datetime. - The values of the histogram can be exported via the
Export graph data
button. A .csv will be saved to[image_directory]/nellie_output/graphs/
with the current datetime. - The histogram's x-axis can be viewed in log10 scale via the
Log scale
checkbox. - By default, the histogram shows lines at the mean +/- 1 standard deviation. This can instead be switched to median and quartiles via the
Median view
checkbox.
- This histogram can be directly exported via the
- Press the
Overlay mask
button to colormap the organelle mask based on your selected feature.- Once overlaid, toggle the
Timepoint data
checkbox to allow you to select a specific timepoint to visualize via the slider.
- Once overlaid, toggle the
Other features¶
- Nellie's plugin offers an
Easy screenshot
feature:- Press the button under
Easy screenshot
or hit Ctrl/Cmd-Shift-E after clicking your image. - The .png will be saved to
[image_directory]/nellie_output/screenshots/
with the current datetime.
- Press the button under
Feedback / Getting Help¶
A few options are available for providing feedback or getting help with Nellie:
Github Issues | email | X | wherever else you can find me!
To avoid any unnecessary back-and-forth, please include any/all (if possible) of the following information in your bug report:
- What kind of computer do you have, and what are its specs?
- Send me screenshots of what is not working.
- Send me any error logs in your terminal.
- Send me the file you ran (if possible).
- Any other information that might be helpful
Other Info¶
For a 16bit dataset, the output:input ratio is ~15x. There is an option in the GUI to automatically delete intermediates after processing, keeping only the CSV files containing the extracted features.
Requirements¶
Nellie has been tested on the following configurations:
- Mac, Linux, and Windows operating systems
- Python >= 3.9
License¶
Nellie © 2024 by Austin E. Y. T. Lefebvre is licensed under CC BY 4.0
Reference¶
If you used Nelly or found this work useful in your own research, please cite our arXiv preprint:
Lefebvre, A. E. Y. T., Sturm, G., et. al. Nellie: Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy, arXiv, 2024, https://arxiv.org/abs/2403.13214
@misc{lefebvre2024nellie,
title={Nellie: Automated organelle segmentation, tracking, and hierarchical feature extraction in 2D/3D live-cell microscopy},
author={Austin E. Y. T. Lefebvre and Gabriel Sturm and Ting-Yu Lin and Emily Stoops and Magdalena Preciado Lopez and Benjamin Kaufmann-Malaga and Kayley Hake},
year={2024},
eprint={2403.13214},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
More fun examples¶
Microtubule growing ends:¶
https://github.com/aelefebv/nellie/assets/26515909/88578dc9-f5c5-4188-a0e2-4e37037a44a9
Endoplasmic reticulum:¶
https://github.com/aelefebv/nellie/assets/26515909/db76d388-a9cc-4650-b93d-69d357ace418
Peroxisomes:¶
https://github.com/aelefebv/nellie/assets/26515909/58bda3cb-6489-4620-8584-a3728cd6b2ec
Code contents:¶
Full documentation can be found within the code, and compiled by Sphinx in the file docs/_build/html/index.html
Nellie pipeline¶
All the Nellie pipeline code is found within the nellie folder
- File and metadata loading, and file preparation is found at nellie/im_info/verifier.py
- Preprocessing is found at nellie/segmentation/filtering.py
- Segmentation of organelles is found at nellie/segmentation/labelling.py
- Skeletonization and segmentation of branches is found at nellie/segmentation/networking.py
- Mocap marker detection is found at nellie/segmentation/mocap_marking.py
- Mocap marker tracking is found at nellie/tracking/hu_tracking.py
- Voxel reassignment via flow interpolation is found at nellie/tracking/voxel_reassignment.py
- Hierarchical feature extraction is found at nellie/feature_extraction/hierarchical.py
Nellie Napari plugin¶
All the Napari plugin code is found with the nellie_napari folder
- The home tab is found at nellie_napari/nellie_home.py
- The file selection tab is found at nellie_napari/nellie_fileselect.py
- The processing tab is found at nellie_napari/nellie_processor.py
- The visualization tab is found at nellie_napari/nellie_visualizer.py
- The analysis tab is found at nellie_napari/nellie_analysis.py
- The settings tab is found at nellie_napari/nellie_settings.py
Version:
- 0.4.0
Last updated:
- 03 October 2024
First released:
- 21 March 2024
License:
- Information not submitted
Supported data:
- Information not submitted
Plugin type:
GitHub activity:
- Stars: 31
- Forks: 4
- Issues + PRs: 2
GitHub activity:
- Stars: 31
- Forks: 4
- Issues + PRs: 2
Operating system:
- Information not submitted
Requirements:
- numpy==1.26.4
- scipy==1.12.0
- scikit-image==0.22.0
- nd2==0.9.0
- ome-types==0.5.2
- pandas==2.2.1
- matplotlib==3.8.3
- napari[all]
- imagecodecs
- pydantic==2.9.2
- pydantic-core==2.23.4