Note

Go to the end to download the full example code.

Loading MRC files (and other binary files)#

This is a simple example of how to load MRC files using Pyxem. The MRC file format is a common format for electron microscopy data. It is a binary format that is used for storing 3D data, such as electron tomography but because it is a fairly simple format, it has been adopted in some cases to store 4D STEM data as well.

First we will download a sample MRC file from the Pyxem data repository. This is a good way to host data if you want to share it with others. I love putting small versions (up to 50 GB) of every dataset I publish on Zenodo and then using pooch to automate the download/extraction process.

import os
import zipfile
import pooch

current_directory = os.getcwd()
file_path = pooch.retrieve(
    # URL to one of Pooch's test files
    url="https://zenodo.org/records/15490547/files/ZrNbMrc.zip",
    known_hash="md5:eeac29aee5622972daa86a394a8c1d5c",
    progressbar=True,
    path=current_directory,
)
# Unzip the file
with zipfile.ZipFile(file_path, "r") as zip_ref:
    zip_ref.extractall(current_directory)

  0%|                                              | 0.00/35.5M [00:00<?, ?B/s]
  0%|                                      | 14.3k/35.5M [00:00<04:33, 130kB/s]
  0%|                                      | 36.9k/35.5M [00:00<03:19, 178kB/s]
  0%|                                       | 104k/35.5M [00:00<01:30, 391kB/s]
  1%|▏                                      | 211k/35.5M [00:00<00:56, 620kB/s]
  1%|▎                                      | 328k/35.5M [00:00<00:44, 793kB/s]
  1%|▍                                      | 442k/35.5M [00:00<00:39, 883kB/s]
  2%|▌                                      | 540k/35.5M [00:00<00:40, 870kB/s]
  2%|▋                                      | 638k/35.5M [00:00<00:39, 891kB/s]
  2%|▊                                      | 728k/35.5M [00:00<00:39, 884kB/s]
  2%|▉                                      | 818k/35.5M [00:01<00:39, 880kB/s]
  3%|▉                                      | 906k/35.5M [00:01<00:39, 872kB/s]
  3%|█                                      | 994k/35.5M [00:01<00:39, 865kB/s]
  3%|█▏                                    | 1.08M/35.5M [00:01<00:40, 858kB/s]
  3%|█▎                                   | 1.23M/35.5M [00:01<00:33, 1.02MB/s]
  4%|█▍                                   | 1.33M/35.5M [00:01<00:33, 1.01MB/s]
  4%|█▍                                   | 1.43M/35.5M [00:01<00:34, 1.00MB/s]
  4%|█▋                                   | 1.57M/35.5M [00:01<00:30, 1.10MB/s]
  5%|█▊                                   | 1.73M/35.5M [00:01<00:27, 1.25MB/s]
  5%|█▉                                   | 1.86M/35.5M [00:02<00:27, 1.23MB/s]
  6%|██                                   | 1.98M/35.5M [00:02<00:27, 1.22MB/s]
  6%|██▏                                  | 2.11M/35.5M [00:02<00:27, 1.21MB/s]
  6%|██▎                                  | 2.26M/35.5M [00:02<00:25, 1.29MB/s]
  7%|██▌                                  | 2.40M/35.5M [00:02<00:24, 1.33MB/s]
  7%|██▋                                  | 2.54M/35.5M [00:02<00:25, 1.29MB/s]
  8%|██▊                                  | 2.67M/35.5M [00:02<00:26, 1.24MB/s]
  8%|██▉                                  | 2.79M/35.5M [00:02<00:26, 1.23MB/s]
  8%|███                                  | 2.92M/35.5M [00:02<00:26, 1.21MB/s]
  9%|███▏                                 | 3.04M/35.5M [00:02<00:27, 1.20MB/s]
  9%|███▎                                 | 3.16M/35.5M [00:03<00:27, 1.19MB/s]
  9%|███▍                                 | 3.28M/35.5M [00:03<00:27, 1.18MB/s]
 10%|███▋                                  | 3.40M/35.5M [00:03<00:44, 719kB/s]
 10%|███▋                                  | 3.50M/35.5M [00:03<00:41, 776kB/s]
 10%|███▊                                  | 3.60M/35.5M [00:03<00:39, 811kB/s]
 10%|███▉                                  | 3.69M/35.5M [00:03<00:38, 837kB/s]
 11%|████                                  | 3.79M/35.5M [00:03<00:37, 856kB/s]
 11%|████▏                                 | 3.89M/35.5M [00:04<00:35, 899kB/s]
 11%|████▎                                 | 3.99M/35.5M [00:04<00:34, 907kB/s]
 11%|████▎                                 | 4.08M/35.5M [00:04<00:34, 910kB/s]
 12%|████▍                                 | 4.18M/35.5M [00:04<00:34, 910kB/s]
 12%|████▌                                 | 4.27M/35.5M [00:04<00:34, 907kB/s]
 12%|████▋                                 | 4.36M/35.5M [00:04<00:34, 902kB/s]
 13%|████▊                                 | 4.46M/35.5M [00:04<00:34, 899kB/s]
 13%|████▊                                 | 4.55M/35.5M [00:04<00:34, 894kB/s]
 13%|████▉                                 | 4.64M/35.5M [00:04<00:35, 880kB/s]
 13%|█████                                 | 4.73M/35.5M [00:04<00:35, 874kB/s]
 14%|█████▏                                | 4.81M/35.5M [00:05<00:35, 866kB/s]
 14%|█████▏                                | 4.90M/35.5M [00:05<00:35, 859kB/s]
 14%|█████▎                                | 4.99M/35.5M [00:05<00:35, 849kB/s]
 14%|█████▍                                | 5.07M/35.5M [00:05<00:36, 842kB/s]
 15%|█████▌                                | 5.16M/35.5M [00:05<00:36, 835kB/s]
 15%|█████▌                                | 5.24M/35.5M [00:05<00:36, 828kB/s]
 15%|█████▋                                | 5.32M/35.5M [00:05<00:36, 820kB/s]
 15%|█████▋                               | 5.48M/35.5M [00:05<00:29, 1.02MB/s]
 16%|█████▊                               | 5.58M/35.5M [00:05<00:29, 1.01MB/s]
 16%|██████                                | 5.68M/35.5M [00:05<00:30, 987kB/s]
 16%|██████▏                               | 5.78M/35.5M [00:06<00:30, 978kB/s]
 17%|██████▎                               | 5.88M/35.5M [00:06<00:30, 969kB/s]
 17%|██████▍                               | 5.98M/35.5M [00:06<00:39, 741kB/s]
 17%|██████▍                               | 6.06M/35.5M [00:06<00:38, 756kB/s]
 17%|██████▌                               | 6.14M/35.5M [00:06<00:38, 764kB/s]
 18%|██████▋                               | 6.22M/35.5M [00:06<00:38, 769kB/s]
 18%|██████▊                               | 6.34M/35.5M [00:06<00:33, 883kB/s]
 18%|██████▉                               | 6.44M/35.5M [00:06<00:33, 879kB/s]
 18%|██████▉                               | 6.53M/35.5M [00:07<00:33, 875kB/s]
 19%|███████                               | 6.62M/35.5M [00:07<00:33, 865kB/s]
 19%|███████▏                              | 6.71M/35.5M [00:07<00:33, 864kB/s]
 19%|███████▎                              | 6.79M/35.5M [00:07<00:33, 851kB/s]
 19%|███████▎                              | 6.88M/35.5M [00:07<00:33, 846kB/s]
 20%|███████▍                              | 6.96M/35.5M [00:07<00:34, 838kB/s]
 20%|███████▌                              | 7.05M/35.5M [00:07<00:34, 833kB/s]
 20%|███████▋                              | 7.13M/35.5M [00:07<00:34, 827kB/s]
 20%|███████▋                              | 7.23M/35.5M [00:07<00:33, 855kB/s]
 21%|███████▊                              | 7.36M/35.5M [00:07<00:28, 975kB/s]
 21%|███████▉                              | 7.46M/35.5M [00:08<00:29, 966kB/s]
 21%|████████                              | 7.56M/35.5M [00:08<00:29, 961kB/s]
 22%|████████▏                             | 7.65M/35.5M [00:08<00:29, 951kB/s]
 22%|████████▎                             | 7.75M/35.5M [00:08<00:29, 942kB/s]
 22%|████████▍                             | 7.84M/35.5M [00:08<00:29, 932kB/s]
 22%|████████▌                             | 7.95M/35.5M [00:08<00:28, 961kB/s]
 23%|████████▌                             | 8.04M/35.5M [00:08<00:28, 951kB/s]
 23%|████████▋                             | 8.14M/35.5M [00:08<00:28, 956kB/s]
 23%|████████▊                             | 8.24M/35.5M [00:08<00:28, 954kB/s]
 23%|████████▉                             | 8.34M/35.5M [00:08<00:28, 946kB/s]
 24%|█████████                             | 8.45M/35.5M [00:09<00:27, 995kB/s]
 24%|█████████▏                            | 8.55M/35.5M [00:09<00:27, 966kB/s]
 24%|█████████▎                            | 8.65M/35.5M [00:09<00:29, 914kB/s]
 25%|█████████▏                           | 8.78M/35.5M [00:09<00:26, 1.01MB/s]
 25%|█████████▎                           | 8.93M/35.5M [00:09<00:23, 1.13MB/s]
 25%|█████████▍                           | 9.04M/35.5M [00:09<00:23, 1.12MB/s]
 26%|█████████▌                           | 9.16M/35.5M [00:09<00:23, 1.11MB/s]
 26%|█████████▋                           | 9.27M/35.5M [00:09<00:23, 1.11MB/s]
 26%|██████████                            | 9.38M/35.5M [00:10<00:30, 845kB/s]
 27%|█████████▉                           | 9.55M/35.5M [00:10<00:25, 1.03MB/s]
 27%|██████████                           | 9.66M/35.5M [00:10<00:24, 1.05MB/s]
 28%|██████████▏                          | 9.78M/35.5M [00:10<00:24, 1.06MB/s]
 28%|██████████▎                          | 9.89M/35.5M [00:10<00:23, 1.07MB/s]
 28%|██████████▋                           | 10.0M/35.5M [00:10<00:30, 836kB/s]
 28%|██████████▊                           | 10.1M/35.5M [00:10<00:27, 914kB/s]
 29%|██████████▉                           | 10.2M/35.5M [00:10<00:27, 930kB/s]
 29%|███████████                           | 10.3M/35.5M [00:10<00:25, 969kB/s]
 29%|██████████▉                          | 10.4M/35.5M [00:11<00:24, 1.01MB/s]
 30%|██████████▉                          | 10.6M/35.5M [00:11<00:24, 1.01MB/s]
 30%|███████████                          | 10.7M/35.5M [00:11<00:24, 1.01MB/s]
 30%|███████████▏                         | 10.8M/35.5M [00:11<00:24, 1.01MB/s]
 31%|███████████▎                         | 10.9M/35.5M [00:11<00:24, 1.00MB/s]
 31%|███████████▋                          | 11.0M/35.5M [00:11<00:24, 995kB/s]
 31%|███████████▊                          | 11.1M/35.5M [00:11<00:32, 761kB/s]
 31%|███████████▉                          | 11.2M/35.5M [00:11<00:31, 776kB/s]
 32%|████████████                          | 11.2M/35.5M [00:12<00:31, 782kB/s]
 32%|████████████                          | 11.3M/35.5M [00:12<00:30, 787kB/s]
 32%|████████████▏                         | 11.4M/35.5M [00:12<00:30, 792kB/s]
 32%|████████████▎                         | 11.5M/35.5M [00:12<00:27, 869kB/s]
 33%|████████████▍                         | 11.6M/35.5M [00:12<00:27, 866kB/s]
 33%|████████████▌                         | 11.7M/35.5M [00:12<00:27, 863kB/s]
 33%|████████████▌                         | 11.8M/35.5M [00:12<00:27, 859kB/s]
 33%|████████████▋                         | 11.9M/35.5M [00:12<00:27, 854kB/s]
 34%|████████████▊                         | 12.0M/35.5M [00:12<00:27, 847kB/s]
 34%|████████████▉                         | 12.0M/35.5M [00:12<00:27, 842kB/s]
 34%|████████████▉                         | 12.1M/35.5M [00:13<00:28, 835kB/s]
 34%|█████████████                         | 12.2M/35.5M [00:13<00:28, 829kB/s]
 35%|█████████████▏                        | 12.3M/35.5M [00:13<00:28, 823kB/s]
 35%|█████████████▏                        | 12.4M/35.5M [00:13<00:28, 817kB/s]
 35%|█████████████▎                        | 12.5M/35.5M [00:13<00:28, 809kB/s]
 35%|█████████████▍                        | 12.5M/35.5M [00:13<00:28, 804kB/s]
 36%|█████████████▍                        | 12.6M/35.5M [00:13<00:28, 798kB/s]
 36%|█████████████▌                        | 12.7M/35.5M [00:13<00:28, 790kB/s]
 36%|█████████████▋                        | 12.8M/35.5M [00:13<00:28, 785kB/s]
 36%|█████████████▊                        | 12.9M/35.5M [00:13<00:29, 778kB/s]
 36%|█████████████▊                        | 12.9M/35.5M [00:14<00:29, 769kB/s]
 37%|█████████████▉                        | 13.0M/35.5M [00:14<00:29, 761kB/s]
 37%|██████████████                        | 13.1M/35.5M [00:14<00:29, 756kB/s]
 37%|██████████████▏                       | 13.2M/35.5M [00:14<00:25, 888kB/s]
 37%|██████████████▏                       | 13.3M/35.5M [00:14<00:25, 879kB/s]
 38%|██████████████▎                       | 13.4M/35.5M [00:14<00:22, 974kB/s]
 38%|██████████████▍                       | 13.5M/35.5M [00:14<00:22, 965kB/s]
 38%|██████████████▏                      | 13.6M/35.5M [00:14<00:21, 1.01MB/s]
 39%|██████████████▋                       | 13.7M/35.5M [00:14<00:21, 998kB/s]
 39%|██████████████▊                       | 13.8M/35.5M [00:15<00:21, 989kB/s]
 39%|██████████████▉                       | 13.9M/35.5M [00:15<00:22, 980kB/s]
 40%|███████████████                       | 14.0M/35.5M [00:15<00:22, 971kB/s]
 40%|███████████████                       | 14.1M/35.5M [00:15<00:28, 741kB/s]
 40%|███████████████▎                      | 14.3M/35.5M [00:15<00:23, 892kB/s]
 40%|███████████████▍                      | 14.4M/35.5M [00:15<00:23, 907kB/s]
 41%|███████████████▍                      | 14.5M/35.5M [00:15<00:22, 916kB/s]
 41%|███████████████▌                      | 14.6M/35.5M [00:15<00:21, 973kB/s]
 41%|███████████████▋                      | 14.7M/35.5M [00:15<00:21, 973kB/s]
 42%|███████████████▊                      | 14.8M/35.5M [00:16<00:21, 972kB/s]
 42%|███████████████▉                      | 14.9M/35.5M [00:16<00:21, 968kB/s]
 42%|████████████████                      | 15.0M/35.5M [00:16<00:21, 960kB/s]
 42%|████████████████▏                     | 15.1M/35.5M [00:16<00:21, 954kB/s]
 43%|████████████████▏                     | 15.2M/35.5M [00:16<00:28, 723kB/s]
 43%|████████████████▎                     | 15.3M/35.5M [00:16<00:27, 738kB/s]
 43%|████████████████▍                     | 15.3M/35.5M [00:16<00:26, 749kB/s]
 43%|████████████████▍                     | 15.4M/35.5M [00:16<00:26, 754kB/s]
 44%|████████████████▌                     | 15.5M/35.5M [00:17<00:26, 757kB/s]
 44%|████████████████▋                     | 15.6M/35.5M [00:17<00:26, 758kB/s]
 44%|████████████████▋                     | 15.7M/35.5M [00:17<00:26, 756kB/s]
 44%|████████████████▊                     | 15.7M/35.5M [00:17<00:25, 765kB/s]
 45%|████████████████▉                     | 15.8M/35.5M [00:18<01:33, 210kB/s]
 45%|████████████████▉                     | 15.9M/35.5M [00:18<01:20, 244kB/s]
 45%|█████████████████                     | 15.9M/35.5M [00:18<01:09, 282kB/s]
 45%|█████████████████                     | 16.0M/35.5M [00:18<01:00, 322kB/s]
 45%|█████████████████▏                    | 16.1M/35.5M [00:18<00:52, 374kB/s]
 45%|█████████████████▎                    | 16.2M/35.5M [00:18<00:39, 491kB/s]
 46%|█████████████████▎                    | 16.2M/35.5M [00:19<00:34, 559kB/s]
 46%|█████████████████▍                    | 16.3M/35.5M [00:19<00:31, 615kB/s]
 46%|█████████████████▌                    | 16.4M/35.5M [00:19<00:29, 658kB/s]
 46%|█████████████████▋                    | 16.5M/35.5M [00:19<00:27, 694kB/s]
 47%|█████████████████▋                    | 16.6M/35.5M [00:19<00:26, 721kB/s]
 47%|█████████████████▊                    | 16.7M/35.5M [00:19<00:21, 875kB/s]
 47%|█████████████████▉                    | 16.8M/35.5M [00:19<00:21, 879kB/s]
 48%|██████████████████                    | 16.9M/35.5M [00:19<00:21, 879kB/s]
 48%|██████████████████▏                   | 17.0M/35.5M [00:19<00:21, 876kB/s]
 48%|██████████████████▎                   | 17.1M/35.5M [00:19<00:21, 874kB/s]
 48%|██████████████████▎                   | 17.1M/35.5M [00:20<00:21, 871kB/s]
 49%|██████████████████▍                   | 17.2M/35.5M [00:20<00:21, 865kB/s]
 49%|██████████████████▌                   | 17.3M/35.5M [00:20<00:21, 857kB/s]
 49%|██████████████████▋                   | 17.4M/35.5M [00:20<00:21, 853kB/s]
 49%|██████████████████▋                   | 17.5M/35.5M [00:20<00:21, 846kB/s]
 49%|██████████████████▊                   | 17.6M/35.5M [00:20<00:21, 839kB/s]
 50%|██████████████████▉                   | 17.7M/35.5M [00:20<00:21, 830kB/s]
 50%|██████████████████▉                   | 17.8M/35.5M [00:20<00:21, 822kB/s]
 50%|███████████████████                   | 17.8M/35.5M [00:20<00:22, 771kB/s]
 50%|███████████████████▏                  | 17.9M/35.5M [00:20<00:23, 765kB/s]
 51%|███████████████████▎                  | 18.0M/35.5M [00:21<00:21, 802kB/s]
 51%|███████████████████▎                  | 18.1M/35.5M [00:21<00:21, 797kB/s]
 51%|███████████████████▌                  | 18.2M/35.5M [00:21<00:17, 979kB/s]
 52%|███████████████████▌                  | 18.3M/35.5M [00:21<00:17, 969kB/s]
 52%|███████████████████▋                  | 18.4M/35.5M [00:21<00:17, 964kB/s]
 52%|███████████████████▊                  | 18.5M/35.5M [00:21<00:17, 956kB/s]
 52%|███████████████████▉                  | 18.6M/35.5M [00:21<00:17, 949kB/s]
 53%|████████████████████                  | 18.7M/35.5M [00:21<00:17, 940kB/s]
 53%|████████████████████▏                 | 18.8M/35.5M [00:21<00:17, 930kB/s]
 53%|████████████████████▏                 | 18.9M/35.5M [00:22<00:18, 917kB/s]
 53%|████████████████████▎                 | 19.0M/35.5M [00:22<00:18, 909kB/s]
 54%|████████████████████▍                 | 19.1M/35.5M [00:22<00:18, 901kB/s]
 54%|████████████████████▌                 | 19.2M/35.5M [00:22<00:18, 892kB/s]
 54%|████████████████████▌                 | 19.3M/35.5M [00:22<00:18, 885kB/s]
 54%|████████████████████▋                 | 19.4M/35.5M [00:22<00:18, 879kB/s]
 55%|████████████████████▊                 | 19.4M/35.5M [00:22<00:18, 871kB/s]
 55%|████████████████████▉                 | 19.5M/35.5M [00:22<00:18, 866kB/s]
 55%|████████████████████▉                 | 19.6M/35.5M [00:22<00:18, 861kB/s]
 56%|█████████████████████                 | 19.7M/35.5M [00:22<00:16, 935kB/s]
 56%|█████████████████████▏                | 19.8M/35.5M [00:23<00:16, 927kB/s]
 56%|█████████████████████▎                | 19.9M/35.5M [00:23<00:17, 912kB/s]
 56%|████████████████████▉                | 20.1M/35.5M [00:23<00:14, 1.04MB/s]
 57%|█████████████████████                | 20.2M/35.5M [00:23<00:14, 1.03MB/s]
 57%|█████████████████████                | 20.3M/35.5M [00:23<00:14, 1.02MB/s]
 57%|█████████████████████▏               | 20.4M/35.5M [00:23<00:14, 1.01MB/s]
 58%|█████████████████████▎               | 20.5M/35.5M [00:23<00:15, 1.00MB/s]
 58%|██████████████████████                | 20.6M/35.5M [00:23<00:15, 992kB/s]
 58%|██████████████████████                | 20.7M/35.5M [00:23<00:15, 982kB/s]
 58%|██████████████████████▏               | 20.8M/35.5M [00:24<00:19, 749kB/s]
 59%|██████████████████████▎               | 20.9M/35.5M [00:24<00:18, 781kB/s]
 59%|██████████████████████▍               | 21.0M/35.5M [00:24<00:18, 790kB/s]
 59%|██████████████████████▌               | 21.0M/35.5M [00:24<00:18, 796kB/s]
 59%|██████████████████████▌               | 21.1M/35.5M [00:24<00:17, 822kB/s]
 60%|██████████████████████▊               | 21.3M/35.5M [00:24<00:14, 993kB/s]
 60%|██████████████████████▊               | 21.4M/35.5M [00:24<00:14, 990kB/s]
 60%|██████████████████████▉               | 21.5M/35.5M [00:24<00:14, 986kB/s]
 61%|███████████████████████               | 21.6M/35.5M [00:24<00:14, 980kB/s]
 61%|███████████████████████▏              | 21.7M/35.5M [00:25<00:14, 974kB/s]
 61%|███████████████████████▎              | 21.8M/35.5M [00:25<00:18, 746kB/s]
 62%|███████████████████████▍              | 21.9M/35.5M [00:25<00:17, 770kB/s]
 62%|███████████████████████▍              | 22.0M/35.5M [00:25<00:16, 816kB/s]
 62%|███████████████████████▌              | 22.1M/35.5M [00:25<00:15, 895kB/s]
 63%|███████████████████████▊              | 22.2M/35.5M [00:25<00:13, 995kB/s]
 63%|███████████████████████▊              | 22.3M/35.5M [00:25<00:13, 993kB/s]
 63%|███████████████████████▉              | 22.4M/35.5M [00:25<00:13, 992kB/s]
 63%|████████████████████████              | 22.5M/35.5M [00:25<00:13, 988kB/s]
 64%|████████████████████████▏             | 22.6M/35.5M [00:26<00:13, 980kB/s]
 64%|████████████████████████▎             | 22.7M/35.5M [00:26<00:17, 752kB/s]
 64%|████████████████████████▍             | 22.8M/35.5M [00:26<00:16, 766kB/s]
 65%|████████████████████████▌             | 22.9M/35.5M [00:26<00:14, 890kB/s]
 65%|████████████████████████▋             | 23.0M/35.5M [00:26<00:13, 900kB/s]
 65%|████████████████████████▊             | 23.1M/35.5M [00:26<00:12, 960kB/s]
 66%|████████████████████████▏            | 23.3M/35.5M [00:26<00:11, 1.04MB/s]
 66%|████████████████████████▎            | 23.4M/35.5M [00:26<00:11, 1.04MB/s]
 66%|████████████████████████▍            | 23.5M/35.5M [00:27<00:11, 1.03MB/s]
 66%|████████████████████████▌            | 23.6M/35.5M [00:27<00:11, 1.05MB/s]
 67%|████████████████████████▋            | 23.7M/35.5M [00:27<00:11, 1.04MB/s]
 67%|████████████████████████▊            | 23.8M/35.5M [00:27<00:11, 1.03MB/s]
 67%|████████████████████████▉            | 23.9M/35.5M [00:27<00:11, 1.02MB/s]
 68%|█████████████████████████            | 24.0M/35.5M [00:27<00:11, 1.01MB/s]
 68%|█████████████████████████            | 24.1M/35.5M [00:27<00:11, 1.01MB/s]
 68%|█████████████████████████▉            | 24.2M/35.5M [00:27<00:14, 769kB/s]
 69%|██████████████████████████            | 24.3M/35.5M [00:27<00:13, 844kB/s]
 69%|██████████████████████████▏           | 24.4M/35.5M [00:28<00:12, 871kB/s]
 69%|█████████████████████████▌           | 24.6M/35.5M [00:28<00:10, 1.02MB/s]
 69%|█████████████████████████▋           | 24.7M/35.5M [00:28<00:10, 1.02MB/s]
 70%|█████████████████████████▊           | 24.8M/35.5M [00:28<00:10, 1.02MB/s]
 70%|█████████████████████████▉           | 24.9M/35.5M [00:28<00:10, 1.02MB/s]
 70%|██████████████████████████           | 25.0M/35.5M [00:28<00:10, 1.02MB/s]
 71%|██████████████████████████▏          | 25.1M/35.5M [00:28<00:10, 1.01MB/s]
 71%|██████████████████████████▉           | 25.2M/35.5M [00:28<00:13, 771kB/s]
 71%|███████████████████████████           | 25.3M/35.5M [00:28<00:11, 869kB/s]
 72%|███████████████████████████▏          | 25.4M/35.5M [00:29<00:11, 887kB/s]
 72%|██████████████████████████▋          | 25.6M/35.5M [00:29<00:09, 1.03MB/s]
 72%|██████████████████████████▊          | 25.7M/35.5M [00:29<00:09, 1.03MB/s]
 73%|██████████████████████████▊          | 25.8M/35.5M [00:29<00:09, 1.03MB/s]
 73%|██████████████████████████▉          | 25.9M/35.5M [00:29<00:09, 1.03MB/s]
 73%|███████████████████████████          | 26.0M/35.5M [00:29<00:09, 1.03MB/s]
 74%|███████████████████████████▉          | 26.1M/35.5M [00:29<00:11, 790kB/s]
 74%|████████████████████████████          | 26.2M/35.5M [00:29<00:11, 804kB/s]
 74%|████████████████████████████          | 26.3M/35.5M [00:30<00:11, 813kB/s]
 74%|████████████████████████████▎         | 26.4M/35.5M [00:30<00:09, 922kB/s]
 75%|████████████████████████████▎         | 26.5M/35.5M [00:30<00:09, 925kB/s]
 75%|████████████████████████████▍         | 26.6M/35.5M [00:30<00:09, 979kB/s]
 75%|████████████████████████████▌         | 26.7M/35.5M [00:30<00:08, 978kB/s]
 76%|████████████████████████████▋         | 26.8M/35.5M [00:30<00:08, 973kB/s]
 76%|████████████████████████████▊         | 26.9M/35.5M [00:30<00:08, 968kB/s]
 76%|████████████████████████████▉         | 27.0M/35.5M [00:30<00:08, 963kB/s]
 76%|█████████████████████████████         | 27.1M/35.5M [00:30<00:08, 957kB/s]
 77%|█████████████████████████████         | 27.2M/35.5M [00:30<00:08, 947kB/s]
 77%|█████████████████████████████▏        | 27.3M/35.5M [00:31<00:08, 939kB/s]
 77%|█████████████████████████████▎        | 27.4M/35.5M [00:31<00:08, 929kB/s]
 77%|█████████████████████████████▍        | 27.5M/35.5M [00:31<00:08, 941kB/s]
 78%|█████████████████████████████▌        | 27.6M/35.5M [00:31<00:08, 928kB/s]
 78%|█████████████████████████████▋        | 27.7M/35.5M [00:31<00:08, 937kB/s]
 78%|████████████████████████████▉        | 27.8M/35.5M [00:31<00:07, 1.03MB/s]
 79%|█████████████████████████████        | 27.9M/35.5M [00:31<00:07, 1.02MB/s]
 79%|█████████████████████████████▏       | 28.0M/35.5M [00:31<00:07, 1.01MB/s]
 79%|█████████████████████████████▎       | 28.1M/35.5M [00:31<00:07, 1.00MB/s]
 80%|██████████████████████████████▏       | 28.2M/35.5M [00:32<00:09, 763kB/s]
 80%|██████████████████████████████▎       | 28.3M/35.5M [00:32<00:09, 777kB/s]
 80%|██████████████████████████████▍       | 28.5M/35.5M [00:32<00:07, 899kB/s]
 80%|██████████████████████████████▌       | 28.6M/35.5M [00:32<00:07, 906kB/s]
 81%|██████████████████████████████▋       | 28.6M/35.5M [00:32<00:07, 910kB/s]
 81%|██████████████████████████████▋       | 28.7M/35.5M [00:32<00:07, 911kB/s]
 81%|██████████████████████████████▊       | 28.8M/35.5M [00:32<00:07, 910kB/s]
 81%|██████████████████████████████▉       | 28.9M/35.5M [00:32<00:07, 906kB/s]
 82%|███████████████████████████████       | 29.0M/35.5M [00:32<00:07, 900kB/s]
 82%|███████████████████████████████▏      | 29.1M/35.5M [00:33<00:07, 893kB/s]
 82%|███████████████████████████████▏      | 29.2M/35.5M [00:33<00:07, 884kB/s]
 82%|███████████████████████████████▎      | 29.3M/35.5M [00:33<00:07, 876kB/s]
 83%|███████████████████████████████▍      | 29.4M/35.5M [00:33<00:07, 865kB/s]
 83%|███████████████████████████████▌      | 29.5M/35.5M [00:33<00:07, 856kB/s]
 83%|███████████████████████████████▌      | 29.6M/35.5M [00:33<00:07, 829kB/s]
 83%|███████████████████████████████▋      | 29.6M/35.5M [00:33<00:07, 824kB/s]
 84%|███████████████████████████████▊      | 29.7M/35.5M [00:33<00:06, 848kB/s]
 84%|███████████████████████████████▉      | 29.8M/35.5M [00:33<00:06, 838kB/s]
 84%|███████████████████████████████▏     | 30.0M/35.5M [00:33<00:05, 1.00MB/s]
 85%|████████████████████████████████▏     | 30.1M/35.5M [00:34<00:05, 996kB/s]
 85%|████████████████████████████████▎     | 30.2M/35.5M [00:34<00:05, 986kB/s]
 85%|████████████████████████████████▎     | 30.3M/35.5M [00:34<00:05, 978kB/s]
 85%|████████████████████████████████▍     | 30.4M/35.5M [00:34<00:06, 747kB/s]
 86%|████████████████████████████████▌     | 30.4M/35.5M [00:34<00:06, 761kB/s]
 86%|████████████████████████████████▋     | 30.5M/35.5M [00:34<00:06, 771kB/s]
 86%|████████████████████████████████▋     | 30.6M/35.5M [00:34<00:06, 776kB/s]
 86%|████████████████████████████████▊     | 30.7M/35.5M [00:34<00:06, 777kB/s]
 87%|████████████████████████████████▉     | 30.8M/35.5M [00:35<00:05, 801kB/s]
 87%|█████████████████████████████████     | 30.9M/35.5M [00:35<00:04, 979kB/s]
 87%|█████████████████████████████████▏    | 31.0M/35.5M [00:35<00:04, 974kB/s]
 88%|█████████████████████████████████▎    | 31.1M/35.5M [00:35<00:04, 969kB/s]
 88%|█████████████████████████████████▍    | 31.2M/35.5M [00:35<00:04, 893kB/s]
 88%|█████████████████████████████████▍    | 31.3M/35.5M [00:35<00:05, 741kB/s]
 88%|█████████████████████████████████▌    | 31.4M/35.5M [00:35<00:05, 748kB/s]
 89%|█████████████████████████████████▋    | 31.5M/35.5M [00:35<00:04, 864kB/s]
 89%|█████████████████████████████████▊    | 31.6M/35.5M [00:35<00:04, 884kB/s]
 89%|█████████████████████████████████▉    | 31.7M/35.5M [00:36<00:04, 885kB/s]
 90%|██████████████████████████████████    | 31.8M/35.5M [00:36<00:04, 882kB/s]
 90%|██████████████████████████████████    | 31.9M/35.5M [00:36<00:04, 878kB/s]
 90%|██████████████████████████████████▏   | 32.0M/35.5M [00:36<00:03, 957kB/s]
 90%|██████████████████████████████████▎   | 32.1M/35.5M [00:36<00:03, 950kB/s]
 91%|██████████████████████████████████▍   | 32.2M/35.5M [00:36<00:03, 996kB/s]
 91%|█████████████████████████████████▋   | 32.4M/35.5M [00:36<00:02, 1.07MB/s]
 91%|█████████████████████████████████▊   | 32.5M/35.5M [00:36<00:02, 1.07MB/s]
 92%|█████████████████████████████████▉   | 32.6M/35.5M [00:36<00:02, 1.06MB/s]
 92%|██████████████████████████████████   | 32.7M/35.5M [00:37<00:02, 1.05MB/s]
 92%|██████████████████████████████████▏  | 32.8M/35.5M [00:37<00:02, 1.04MB/s]
 93%|██████████████████████████████████▎  | 32.9M/35.5M [00:37<00:02, 1.03MB/s]
 93%|██████████████████████████████████▎  | 33.0M/35.5M [00:37<00:02, 1.02MB/s]
 93%|██████████████████████████████████▍  | 33.1M/35.5M [00:37<00:02, 1.01MB/s]
 93%|███████████████████████████████████▌  | 33.2M/35.5M [00:37<00:02, 999kB/s]
 94%|███████████████████████████████████▌  | 33.3M/35.5M [00:37<00:02, 763kB/s]
 94%|███████████████████████████████████▋  | 33.4M/35.5M [00:37<00:02, 775kB/s]
 94%|███████████████████████████████████▊  | 33.5M/35.5M [00:37<00:02, 785kB/s]
 94%|███████████████████████████████████▉  | 33.5M/35.5M [00:38<00:02, 766kB/s]
 95%|███████████████████████████████████▉  | 33.6M/35.5M [00:38<00:02, 755kB/s]
 95%|████████████████████████████████████  | 33.7M/35.5M [00:38<00:02, 774kB/s]
 95%|████████████████████████████████████▏ | 33.8M/35.5M [00:38<00:02, 820kB/s]
 95%|████████████████████████████████████▎ | 33.9M/35.5M [00:38<00:01, 816kB/s]
 96%|████████████████████████████████████▎ | 34.0M/35.5M [00:38<00:01, 813kB/s]
 96%|████████████████████████████████████▍ | 34.1M/35.5M [00:38<00:01, 805kB/s]
 96%|████████████████████████████████████▌ | 34.1M/35.5M [00:38<00:01, 802kB/s]
 96%|████████████████████████████████████▌ | 34.2M/35.5M [00:38<00:01, 796kB/s]
 97%|████████████████████████████████████▋ | 34.3M/35.5M [00:38<00:01, 788kB/s]
 97%|████████████████████████████████████▊ | 34.4M/35.5M [00:39<00:01, 782kB/s]
 97%|████████████████████████████████████▊ | 34.5M/35.5M [00:39<00:01, 774kB/s]
 97%|████████████████████████████████████▉ | 34.6M/35.5M [00:39<00:01, 838kB/s]
 98%|█████████████████████████████████████ | 34.7M/35.5M [00:39<00:00, 964kB/s]
 98%|█████████████████████████████████████▏| 34.8M/35.5M [00:39<00:00, 955kB/s]
 98%|█████████████████████████████████████▎| 34.9M/35.5M [00:39<00:00, 946kB/s]
 98%|█████████████████████████████████████▍| 35.0M/35.5M [00:39<00:00, 935kB/s]
 99%|█████████████████████████████████████▌| 35.1M/35.5M [00:39<00:00, 921kB/s]
 99%|█████████████████████████████████████▌| 35.2M/35.5M [00:39<00:00, 893kB/s]
 99%|█████████████████████████████████████▋| 35.3M/35.5M [00:40<00:00, 884kB/s]
100%|████████████████████████████████████▊| 35.4M/35.5M [00:40<00:00, 1.01MB/s]
100%|█████████████████████████████████████▉| 35.5M/35.5M [00:40<00:00, 997kB/s]
  0%|                                              | 0.00/35.5M [00:00<?, ?B/s]
100%|██████████████████████████████████████| 35.5M/35.5M [00:00<00:00, 249GB/s]

Loading the MRC file#

We can now load the file using the load method from hyperspy. This method uses the MRC Reader to read the file. In this case, because the file was collected with a Direct Electron camera, the metadata is automatically loaded as well.

import hyperspy.api as hs

signal = hs.load(
    "ZrNbMrc/20241021_00405_movie.mrc",
)

Loading Lazily#

In this case the file was loaded using the numpy.memmap function, this won’t load the entire file into memory, but if for example you do signal.sum() now the entire file will be loaded into memory. In most cases it is better to just use the lazy=True option to load the file lazily.

signal = hs.load("ZrNbMrc/20241021_00405_movie.mrc", lazy=True)

signal
Title:
SignalType:
Array Chunk
Bytes 37.50 MiB 37.50 MiB
Shape (30, 20|128, 128) (30,20|128,128)
Count 2 Tasks 1 Chunks
Type float32 numpy.ndarray

Navigation Axes

Signal Axes
30 20 128 128


Controlling the Chunk Size#

The chunk size is the number of frames that will be loaded into memory at once when the signal is lazy loaded. This can be controlled using the chunks parameter. A good place to start is to use the auto option for the first two dimensions, which will automatically determine the chunk size based on the available memory. The last two dimensions are the reciprocal space dimensions, as we usually map over those dimensions we can set them to -1 to indicate that we want to load all the data in those dimensions at once.

signal = hs.load("ZrNbMrc/20241021_00405_movie.mrc", lazy=True, chunks=(10, 10, -1, -1))

signal
Title:
SignalType:
Array Chunk
Bytes 37.50 MiB 6.25 MiB
Shape (30, 20|128, 128) (10,10|128,128)
Count 12 Tasks 6 Chunks
Type float32 numpy.ndarray

Navigation Axes

Signal Axes
30 20 128 128


Slicing the Signal#

Interestingly, binary files are sometimes faster than compressed formats. With compressed file formats, like HDF5 or Zarr, you need to decompress the entire chunk before you can access and part of the data. For things like Virtual Images or slicing a signal this can add overhead. With binary files, because the underlying data is a memory map, even for dask arrays, you can very efficiently slice parts of the data without loading the entire chunk into memory.

slice_sum = signal.isig[0:10, 0:10].sum()
slice_sum.compute()

  0%|          | 0/16 [00:00<?, ?it/s]
100%|██████████| 16/16 [00:00<00:00, 3030.43it/s]

In this case this is faster than the compressed equivalent, because we don’t have to load the entire chunk into memory just to throw most of it away.

A couple of more things to note. Performance of binary files is usually better on SSDs than on HDDs, because the seek time is much lower on SSDs. This means that you can have arbitrary dask chunks and it will still be fast. On HDDs, you want to keep data that is close together in the same chunk. Usually this means you want chunks like (1, “auto”, -1, -1). This is not terribly noticeable for 1-2 GB files, somewhat noticeable for 10-20 GB files, and extremely important for 100+ GB files on an HDD.

Total running time of the script: (0 minutes 40.951 seconds)

Gallery generated by Sphinx-Gallery