CHANGELOG.rst

DIALS 3.26.0 (2025-11-06)

Features

• Add mcd outlier rejection for stills, and add new options for mcd coordinates (xy vs. radial and transverse) (#2676)
• dials.modify_experiments: new command that replaces dials.modify_geometry and allows changes to crystal models. (#2936)
• dials.import: Expand recursive wildcards (./some/**/*.h5). (#2957)
• Time-of-flight data: Ellipse and Seed Skewness foreground/background masking algorithms added. (#2964)
• Laue spot finding: Add wrapper that adds initial wavelengths to reflections. (#2972)
• dials.index: Enable indexing.method=low_res_spot_match for polychromatic data. (#2984)
• dials.index: Enable PinkIndexer to be used with joint indexing and with polychromatic experiments. (#2985)
• dials.find_spots: Add filter for time-of-flight data, to filter out spots within close proximity in reciprocal space. (#2986)
• Automatically use gzip-compressed reflection tables by setting the environment variable DIALS_USE_GZIP_REFL. (#2987)
• dials.correlation_matrix: Optimise the input parameters to OPTICS, for better detection of clusters. (#2999)
• dials.generate_mask: Added a new option, merge_imageset_mask, to merge pixel mask(s) defined in the ImageSet, such as a mask from the dxtbx class and a mask specified during import. (#3034)
• dials.combine_experiments: Add pathname/image index sorting. (#3036)

Bugfixes

• dials.import: Allow per-shot wavelength variations in convert_stills_to_sequence=true. (#2865)
• dials.index: Do not lose the experiment history dials.export: Always write timestamps in UTC. Also capture processing history lines for dials.integrate and dials.scale for later addition to the MTZ Appendix, or similar. (#2924)
• Time-of-flight: Fix mismatch between calculated values for reflections generated from indexing, and those from integration. (#2968)
• dials.find_spots: No longer write out an empty strong.refl if no reflections are found. (#2969)
• Add extra checks for Scans having oscillation values in centroid_px_to_mm_panel and Indexer._xyzcal_mm_to_px. (#2973)
• Remove beam.get_s0() calls from geometry_viewer and reciprocal lattice viewer, to allow these to be used with experiments containing PolychromaticBeams. (#2981)
• dials.refine_bravais_settings: Only refine the crystal model by default, not the detector and beam too. (#2983)
• dials.export format=xds: Ensure only strong reflections are exported in SPOT.XDS and reverse-sort by intensity. (#2988)
• dials.search_beam_position: Projection methods now work for each detector separately. (#2993)
• dials.stills_process: Bugfix for multiprocessing mode. (#2995)
• dials.split_experiments: Report number of reflections in each output file. (#3015)
• dials.estimate_resolution: Don't output a large plot dict to the end of stdout. (#3018)
• Enable parallel building in bootstrap.py installs on Unix machines. (#3027)
• Fix all search_beam_position projection methods to work with the average diffraction image (#3045)

Improved Documentation

• Update the instructions in the DPF3 Part 2: A question of centring tutorial. (#3006)
• Remove an outdated multi-lattice tutorial that uses BLEND. (#3031)

Deprecations and Removals

• dials.modify_geometry is deprecated (use dials.modify_experiments instead). (#2936)

Misc

• #2949, #2989, #2991, #2997, #3000, #3002, #3003, #3004, #3007, #3013, #3026

DIALS 3.25.0 (2025-06-20)

This release contains a number of new features for multi-crystal processing. - The dials.import_xds program has been significantly updated; given an INTEGRATE.HKL and XDS.INP from XDS processing, one can now convert these to DIALS data files that can be used in xia2.multiplex and other DIALS data reduction tools. See https://dials.github.io/documentation/programs/dials_import_xds.html for the program documentation. - xia2.multiplex now generates unmerged mmcif files that are suitable for deposition of unmerged data. - Improvements have been made to the xia2.multiplex log output (better logging of progress and individual logs for more DIALS programs) and runtime performance has been improved for large datasets.

Features

• dials.reflection_viewer: Report reflection table columns on launch. (#2912)
• dials.index: Allow a trial crystal to be specified by full setting matrix A=UB. (#2939)
• DIALS is now NumPy 2.0+ compatible. (#2942)
• Python 3.13 is now supported, and is the default version in bootstrap.py. (#2946)

Bugfixes

• dials.ssx_integrate: Do not output a reflection file if no reflections were integrated. (#2804)
• Updated test suite to use data from dials-data rather than dials_regression, meaning that much more of the test suite can be run outside of Diamond. (#2907)
• dials.search_beam_position: Fix printing of the beam center. (#2910)
• dials.import_xds: Reworked the import from INTEGRATE.HKL to have a more complete matchup between the data provided in that file and the reflection list expected by DIALS. (#2914)
• dials.index: Avoid test failure when the ffbidx package is not functional. (#2923)
• dials.export: Ensure mmcif output contains the same reflections as mtz output by default. (#2929)
• dials.import_xds: Clarify expected inputs and improve program structure and output. (#2938)
• dials.import_xds: Fill xyzobs column values for xds unobserved reflections (i.e. not strong). (#2950)
• dials.import_xds: Warn the user if XDS.INP is not found, but a case-insensitive variant is present. (#2951)
• Make mmcif citations correct for import_xds and ssx data processing. (#2955)

Deprecations and Removals

• The dials_regression fixture for tests is removed, as no tests use this repository any more. (#2919)

Misc

• #2906, #2915, #2925, #2928, #2932, #2933, #2935, #2937, #2940, #2947, #2952

DIALS 3.24.3 (2025-05-27)

No significant changes.

DIALS 3.24.2 (2025-05-16)

Internal release to fix generation of release-tagged docker images.

DIALS 3.24.1 (2025-05-13)

Bugfixes

• Add missing implementation in multi-dataset-handling OrderedSet. This caused a crash running xia2.ssx_reduce. (#2916)

DIALS 3.24.0 (2025-04-29)

Features

• dials.index: Separately report unindexed reflection counts for reflections away from ice regions. (#2827)
• dials.cosym: Add the option output.excluded=True to set aside the data sets that cosym rejected, so that these can more easily be investigated separately. (#2854)
• dials.image_viewer: Support iso-resolution lines (circles, ellipses, hyperbola) on non-coplanar and/or two_theta != 0 detectors. (#2867)
• dials.correlation_matrix: Hierarchical clustering now defaults to Ward linkages. (#2876)
• dials.cluster_unit_cell: Add option to use Ward linkage, rather than single linkage. (#2884)
• dials.correlation_matrix: Perform principal component analysis on optimised cosym coordinates. Coordinate plots now rotated by principal components. Add a new matrix-style plot for displaying all projections at once. (#2885)

Bugfixes

• Switch from using unmaintained orderedset to ordered_set (#2749)
• dials.find_spots: Prevent masked pixels from causing a "halo" of excluded pixels around them when using threashold.algorithm=extended_dispersion (the default). This previously caused strong pixels to be excluded from module edges and incorrectly classed strong pixels as background if close to masked pixels. (#2790)
• dials.ssx_index: Fix crash when processing stills with no scan objects. (#2841)
• dials.export format=mosflm: Do not write a DIRECTORY line if the image files do not exist (#2845)
• dials.cluster_unit_cell: output each singleton point group and unit cell on its own line. (#2850)
• dials.cluster_unit_cell: Use logging rather than print for output. (#2852)
• dials.refine: Warn, don't crash, if trying to fix individual scan-varying parameters. (#2856)
• dials.cluster_unit_cell: Clusters are given a cluster_id in order of decreasing size, and the output filenames match the cluster_id. (#2859)
• dials.find_bad_pixels: Use dispersion algorithm for filtering, instead of extended dispersion. This is more sensitive for lone pixels. (#2881)
• dials.scale: Fix potential bad scaling against a reference if resolution limit is greater than 2 Angstroms. (#2883)
• dials.reciprocal_lattice_viewer: Fix translate function for Python 3.12. (#2888)
• dials.stills_process: Miscellaneous fixes when using with XTC files. (#2899)

Improved Documentation

• 3DED/MicroED papers added to the website. (#2847)
• dials.export: Revise usage documentation to be more explicit about which experiment files to use for different cases. (#2864)
• The user support mailing list is now dials-user-group@jiscmail.net (#2866)
• Add the user group e-mail address, dials-user-group@jiscmail.ac.uk, to the message shown on errors. (#2893)

Deprecations and Removals

• Remove deprecated use of pkg_resources, which is removed in Python 3.12. (#2855)

Misc

• #2842, #2843, #2857, #2871, #2887, #2892, #2895, #2901, #2903, #2904, #2905

Dials 3.24.0 (2025-04-28)

Features

• dials.cosym: Add the option output.excluded=True to set aside the data sets that cosym rejected, so that these can more easily be investigated separately. (#2854)
• dials.image_viewer: support iso-resolution lines (circles, ellipses, hyperbola) on non-coplanar and/or two_theta != 0 detectors (#2867)
• dials.correlation_matrix: ward linkages now default for hierarchical clustering (tailorable through phil parameter) (#2876)
• dials.cluster_unit_cell: Add option to use ward linkage rather than single linkage (#2884)
• dials.correlation_matrix: Perform principal component analysis on optimised cosym coordinates. Coordinate plots now rotated by principal components. New matrix-style plot for displaying all projections. (#2885)

Bugfixes

• Switch from using unmaintained orderedset to ordered_set (#2749)
• dials.find_spots: Prevent masked pixels from causing a "halo" of excluded pixels around them when using threashold.algorithm=extended_dispersion (the default). This previously caused strong pixels to be excluded from module edges and incorrectly classed strong pixels as background if close to masked pixels. (#2790)
• dials.ssx_index: Fix crash when processing stills with no scan objects. (#2841)
• dials.export format=mosflm: Do not write a DIRECTORY line if the image files do not exist (#2845)
• dials.cluster_unit_cell: output each singleton point group and unit cell on its own line. (#2850)
• dials.cluster_unit_cell: Use logging rather than print for output. (#2852)
• Remove deprecated use of pkg_resources, which is removed in Python 3.12. (#2855)
• dials.refine: warn if trying to fix individual scan-varying parameters rather than crashing (#2856)
• precommitbx: allow install on Python 3.12 (#2857)
• dials.cluster_unit_cell: Clusters are given a cluster_id in order of decreasing size, and the output filenames match the cluster_id. (#2859)
• dials.export: revise usage documentation to be more explicit about which experiment files to use for different cases (#2864)
• Use locally defined OrderedSet for cases where the implementation in ordered_set does not work. (#2871)
• dials.find_bad_pixels: use dispersion not dispersion extended for filtering (#2881)
• dials.scale: Fix potential bad scaling against a reference if resolution limit greater than 2 Angstroms (#2883)
• No longer use files from dials_regression/indexing_test_data/ in tests. The relevant files are now in dials-data. (#2887)
• dials.reciprocal_lattice_viewer: Fix translate function for Python 3.12 (#2888)
• Only use image example data from the publicly-available dials-data. (#2892)
• Add the user group e-mail address, dials-user-group@jiscmail.ac.uk, to the message shown on errors. (#2893)
• Bugfixes for dials.stills_process when used with XTC files (#2899)

Improved Documentation

• 3DED/MicroED papers added to the website. (#2847)
• The user support mailing list is now dials-user-group@jiscmail.net (#2866)

Misc

• #2842, #2843, #2895, #2901, #2903, #2904, #2905

DIALS 3.23.0 (2025-01-08)

Features

• dials.estimate_resolution: Add resolution estimate based on limit of cc1/2 significance. (#2580)
• Added initial support for a H5-based reflection table. This is turned off by default, but can be activated by setting the DIALS_USE_H5 environment variable. (#2677)
• dials.refine_error_model: Allow grouped and individual error model refinements. (#2685)
• dials.symmetry: Enable oversampling in systematic_absences.method=fourier, for screw-axis detection. (#2701)
• dials.index and dials.ssx_index: Add the CUDA-accelerated fast-feedback-indexer to DIALS as a lattice search algorithm. See https://github.com/paulscherrerinstitute/fast-feedback-indexer for more details. (#2717)
• More DIALS tests are changed to use publicly-available data via dials-data (#2758)
• dials.correlation_matrix: Include reachability plot for visualisation of coordinate-clustering. (#2778)
• Python 3.10 is now the minimum python supported. (#2792)
• dials.image_viewer: Add a line tool that gives start, end, and midpoints, plus a line profile graph. (#2798)
• dials.search_beam_position: Include three new projection methods. (#2809)
• dials.import: Add syntax for splitting a long scan into chunks e.g. for sequential data collections on small crystals. New syntax is either split=frames_per_block or split=start,end,frames_per_block, if you wish to specify the range, where start and end are 1-indexed. (#2816)

Bugfixes

• Ensure logger info is always printed correctly in spot-finding. (#2678)
• Fix beam.get_s0() errors when trying to use image viewer with polychromatic data. (#2765)
• Fix symmetry not being applied consistently post indexing, when multiple crystals and multiple orientations are present. (#2786)
• dials.generate_distortion_maps: Correct definitions for creating elliptical distortion correction maps. (#2787)
• dials.predict: Now adds experiment identifiers to output reflections. (#2788)
• dials.image_viewer: Recover missing image scrollbar by increasing the panel height. (#2795)
• dials.index: fix inconsistent experiment IDs after indexing with index_assignment.method=local (#2799)
• dials.index: Fix minor bug in global index assignment. (#2800)
• dials.predict: Do not try to filter shadowed reflections, if no masker is provided. (#2805)
• dials.cosym: Use unbiased formula for standard error. (#2810)
• dials.cosym: Fix weighting of cosym objective function, use count weights by default. (#2813)
• dials.image_viewer: Do not allow integer controls to take values outside of their bounds. (#2822)
• dials.refine_bravais_settings: Select only the reflections used in refinement in the indexing step for the calculation, to reduce memory footprint with very high resolution data. (#2824)
• dials.image_viewer: Fix display bug with the trusted range mask, when viewing stacked images. (#2830)
• dials.export: Fixes to work with gemmi v0.7. (#2834)
• dials.image_viewer: Avoid error messages when quitting while one of the tools (ring, unit cell, etc.) is open. (#2836)
• dials.integrate: Allow command line option d_min= without requiring full qualification as prediction.d_min=. (#2837)
• More fixes for gemmi 0.7. (#2839)

Misc

• #2528, #2579, #2769, #2776, #2779, #2780, #2791, #2794, #2796, #2801, #2802, #2808, #2818, #2821, #2832, #2833, #2838

DIALS 3.22.0 (2024-10-15)

Features

• Added scaling corrections for time-of-flight data. (#2704)
• dials.generate_mask: Allow combining masks. (#2711)
• dials.image_viewer: Added multi-panel polygon masking. (#2735)
• dials.correlation_matrix: Add dimension optimisation for intensity-based dataset clustering. (#2743)
• dials.generate_distortion_maps: Much faster map generation for mode=ellipse. (#2745)
• Switch bootstrap.py to use CMake by default. You can pass --libtbx to get the previous behaviour, if you need to develop cctbx. (#2755)
• dials.correlation_matrix: Add auto-selection and output of significant clusters and extended graphs in html output (#2770)

Bugfixes

• dials.index: Ensure reflections associated with removed experiments are reset to unindexed. (#2653)
• dials.find_spots: Enable threshold.algorithm=radial_profile for polychromatic data. (#2729)
• dials.image_viewer: Fixed stacking of masks, and stacking of multiple experiments. (#2730)
• dials.image_viewer: Fix UI glitches in masking tools. (#2731)
• dials.resolution_analysis: Suppress output of potential scipy OptimizeWarning. (#2737)
• dials.generate_distortion_maps: Fix bug with mode=ellipse for detectors with oblong panels. (#2740)
• dials.cosym: Handle the case of very few reflections, leading to zero weights in weighted cosym analysis. (#2741)
• dials.find_spots_server: fix multiprocessing to use fork on macOS and on all platforms (and needed for Python 3.14) (#2747)
• DIALS viewers: Tweak size of UI elements for GTK3. (#2753)
• dials.integrate: Ensure the polarization correction is only performed for X-ray data (not electron nor neutron). (#2754)
• dials.image_viewer: Fix broken "Save As" PNG functionality. (#2759)
• API: Ensure correct results from the determine_reindex_operator_against_reference function when the test and reference miller sets are the same object. (#2761)
• dials.index: When indexing multiple imagesets, ensure a failure to index one does not result in failure for all. (#2764)
• dials.image_viewer: Fix errors when trying to view polychromatic data. (#2765)
• dials.reciprocal_lattice_viewer: Fix errors when trying to view polychromatic data. (#2766)
• dials.geometry_viewer: Fix errors when trying to view polychromatic experiments. (#2773)

Improved Documentation

• dials.find_spots: Added second implementation of signal mask erosion, for documentation. (#2665)

Deprecations and Removals

• bootstrap.py no longer accepts --prebuilt-cctbx. This was only ever supported in CI. If you want to build a distribution with prebuilt cctbx, please use the --cmake flag and build instead. (#2727)
• dials.find_spots_server: No longer supported in Windows. (#2747)

Misc

• #2727, #2736, #2738, #2742, #2756, #2767, #2768

Dials 3.22.0 (2024-10-15)

Features

• dials.find_spots: add second implementation of signal mask erosion for documentation and clarify comments (#2665)
• Added time-of-flight Lorentz, spherical absorption, and normalisation w.r.t empty and incident run scaling corrections. (#2704)
• Allow combining masks using dials.generate_mask (#2711)
• Implemented multi-panel polygon masking in dials.image_viewer (#2735)
• dials.correlation_matrix: Add dimension optimisation for intensity-based dataset clustering (#2743)
• dials.generate_distortion_maps: Much faster map generation for mode=ellipse. (#2745)
• Make CMake the default build mode of bootstrap. (#2755)
• dials.correlation_matrix: Add auto-selection and output of significant clusters and extended graphs in html output (#2770)

Bugfixes

• dials.index: ensure reflections associated with removed experiments are reset to unindexed. (#2653)
• Enable radial_profile spot finding algorithm for polychromatic data by removing call to beam s0. (#2729)
• Fixed stacking of masks and stacking of multiple experiments (e.g. stills) in dials.image_viewer (#1512, #2724) (#2730)
• dials.image_viewer: Fix UI glitches in masking tools. (#2731)
• dials.resolution_analysis: Suppress output of potential scipy OptimizeWarning. (#2737)
• dials.generate_distortion_maps: fix bug with mode=ellipse for detectors with oblong panels. (#2740)
• dials.cosym: Handle the case of very few reflections leading to zero weights in weighted cosym analysis. (#2741)
• dials.find_spots_server: fix multiprocessing to use fork on macOS and on all platforms (and needed for Python 3.14) (#2747)
• DIALS viewers: tweak size of UI elements. (#2753)
• dials.integrate: Ensure the polarization correction is only performed for X-ray data (not electron nor neutron). (#2754)
• dials.image_viewer: Fix broken "Save As" PNG functionality. (#2759)
• Ensure correct results from the determine_reindex_operator_against_reference function when the test and reference miller sets are the same object. (#2761)
• dials.index: when indexing multiple imagesets, ensure a failure to index one does not result in failure for all. (#2764)
• Fix beam.get_s0() errors when trying to use image viewer with polychromatic data. (#2765)
• Fix reciprocal lattice viewer not working with polychromatic data due to call to beam.get_s0(). (#2766)
• Fix errors with loading polychromatic experiments into dials.geometry_viewer. (#2773)

Deprecations and Removals

• bootstrap.py no longer accepts --prebuilt-cctbx. This was only ever supported in CI. If you want to build a distribution with prebuilt cctbx, please use the --cmake flag and build instead. (#2727)
• dials.find_spots_server: No longer supported in Windows. (#2747)

Misc

• #2727, #2736, #2738, #2742, #2756, #2767, #2768

DIALS 3.21.1 (2024-08-23)

Bugfixes

• CMake build can now be used for release branches. (#2727)

DIALS 3.21.0 (2024-08-20)

Features

• Add classes to support time-of-flight and Laue indexing and refinement. (#2662)
• dials.symmetry: Allow free selection of significance_level in the range [0,1]. (#2696)
• dials.export: Add support for exporting still data in mmcif format that GEMMI can read. (#2709)

Bugfixes

• dials.index: Fix a pink_indexer error that caused failures for images with electron diffraction geometry. (#2680)
• dials.correlation_matrix: Correctly select datasets for output json after filtering, when used by multiplex. (#2681)
• dials.index: Avoid mm to px conversion when the refinement_protocol is set to do no refinement, as the required data are not available. (#2687)
• Avoid deprecated matplotlib.cm.get_cmap calls (#2688)
• Change Docker base image to rockylinux:8 as centos:7 is EOL (#2689)
• dials.image_viewer: Increase the maximum resolution for the ring tool radius. This was too small for new detectors. (#2697)
• dials.index: Fix potential crash in max_cell estimation when all spots are at the resolution of ice rings. (#2699)
• dials.merge: Use GEMMI to output merged MTZs, for consistency with dials.export (#2700)
• dials.reciprocal_lattice_viewer: Fix middle mouse drag to translate function. (#2707)
• Performance improvement for selections from large reflection tables. For a table containing 165k experiment identifiers the speedup is 1000x (12 minutes per call). (#2718)

Improved Documentation

• Docstrings and type hints are added to the reindex_experiments and reindex_reflections functions to make it easier to use these outside the dials.reindex program. (#2683)
• Improvements to the small molecule tutorial. (#2692)

Deprecations and Removals

• dials.import: remove useless as_grid_scan option as misleading. To import as stills set scan.oscillation=0,0 (#2454)

Misc

• #2675, #2702, #2714, #2715, #2723

DIALS 3.20.0 (2024-06-19)

Features

• New tool: dials.correlation_matrix: A new command-line tool for correlation and cosine similarity clustering of multi-crystal datasets, independent of xia2.multiplex. It provides HTML output, including clustering heatmaps, dendrograms and corresponding dials.cosym graphs. (#2632)
• DIALS is now compatible with Python 3.12. (#2651)
• dials.scale: Add filtering options to default basic error model to allow error modelling of stills data. (#2654)
• dials.cosym: Add alternative weighting during cosym CC calculation with a new cc_weights= option. (#2666)
• dials.correlation_matrix: Add tables with cluster information to html output. (#2671)
• New tool: dials.split_still_data for splitting dials-processed still data based on image number (e.g. dose series). (#2672)

Bugfixes

• dials.find_rotation_axis: Correctly set the orientation of the rotation axis for a multi-axis goniometer. (#2658)
• Fix dials.show beam checks for time of flight experiments. (#2660)
• When masking, raise an error if d_min > d_max (where no spots would be found). (#2664)
• dials.cosym: Make function return structure correctly, in the recently added cc_weights= option. (#2668)
• dials.find_rotation_axis: Add reflection selection criteria, to avoid runs that use a very large amount of memory. (#2670)
• dials.cosym: For cc_weights=sigma, ensure correct filtering based on min_pairs parameters. (#2673)
• dials.cosym: Fix to give more accurate cc calculation when running with a space_group= set. (#2674)
• Fix flex.reflection_table.concat, to not modify in-place. (#2679)

Misc

• #2613, #2616, #2641, #2667

DIALS 3.19.1 (2024-05-23)

No significant changes.

DIALS 3.19.0 (2024-04-17)

Features

• dials.predict: Allow usage when image data are not available. (#2553)
• Add TOFSpotFinder to tailor default params to time of flight experiments and add additional reflection table data. (#2602)
• dials.ssx_index: Allow use of sequences indexer, pink_indexer and low_res_spot_match indexing algorithms. (#2614)
• dials.export: Add option composition= to specify of asymmetric unit composition for SHELX .ins file output. (#2623)

Bugfixes

• Fix building on RHEL8 and other more recent distributions. (#1465)
• dials.index: Joint indexing is automatically set on for rotation data, off for still data. This can be overridden by explicit use of joint_indexing=. (#2605)
• dials.export format=shelx: Increased precision of unit cell parameters and their estimated standard deviations written to .ins file. (#2624)
• dials.ssx_index: Don't combine detector models if individually refined. (#2634)
• dials.scale: Fix error in propagation of partiality variance, when scaling still data. (#2642)
• dials.export: Fix crash when exporting scaled still-shot data. (#2646)

Deprecations and Removals

• API: array_family/flex_ext.py: remove nthread``s parameter from ``extract_shoeboxes, as it was never implemented. (#2638)

Misc

• #2617, #2618, #2619, #2626, #2633, #2648

DIALS 3.18.1 (2024-03-26)

Bugfixes

• wxPython was inadvertently reverted to 4.1 instead of 4.2 for python 3.9 and 3.10 releases. (#2636)

DIALS 3.18.0 (2024-02-22)

Features

• dials.slice_sequence: Add exclude_images_multiple= option to split a scan at calibration images, as an alternative to excluding those images in dials.integrate. (#2565)
• dials.index: Add indexing.method=pink_indexer for still images, based on Gevorkov Y et al. pinkIndexer – a universal indexer for pink-beam X-ray and electron diffraction snapshots. (#2604)

Bugfixes

• dials.index: Correctly output imageset id when joint_indexing=False and max_lattices>1 (#2538)
• dials.index: Ensure the unindexed_reflections table in the indexer classes is updated during refinement. (#2545)
• When building a new developer installation without CMake, always defer to environmental flags when configuring compilers. This fixes building on MacOS Sonoma; for which the default libtbx configuration failed. (#2546)
• dials.cosym: Ensure correct reindexing of low symmetry data to the correct setting. (#2548)
• dials.export: Unmerged MTZ output now uses gemmi, fixing various longstanding issues. (#2549)
• dials.symmetry/dials.cosym: Avoid crashes when unable to find consistent symmetry during cell reduction routine. (#2552)
• dials.sequence_to_stills: Avoid creating an experiment for an image with no reflections on it. (#2554)
• dials.reciprocal_lattice_viewer: Fix toggling of ids on spotfinding output when viewing multiple files. (#2566)
• dials.estimate_resolution: Fix bug causing fitting failures for isigma, misigma curves. (#2568)
• dials.reciprocal_lattice_viewer: Ensure the correct panel for the beam centre is recorded when loading models. (#2572)
• dials.reindex: When reindexing against a reference, use the best cell when determining lattice symmetry. (#2573)
• dials.cosym: Filter out datasets that can't be mapped through a consistent best cell, rather than exiting. (#2574)
• dials.export_bitmaps: Fix for use of imageset_index parameter on image sets that don't start at image numbers of 1. (#2591)
• Reorganize imports in export_mtz.py so that its gemmi-less objects can be used by other gemmi-less software. (#2592)
• dials.image_viewer: Fix for stills experiments viewing crash. (#2594)
• dials.refine: Clarify a misleading error message when there are no reflections to refine. (#2607)
• Dependency update for cctbx/dxtbx#697: adds natsort (#2611)

Improved Documentation

• dials.refine: Update online documentation. (#2551)

Misc

• #2408, #2547, #2550, #2556, #2557, #2563, #2570, #2576, #2578, #2586, #2590, #2599, #2600, #2603

Dials 3.18 (2024-02-22)

Features

• dials.slice_sequence: Add exclude_images_multiple option to split a scan at calibration images, as an alternative to excluding those images in dials.integrate. See #1992 for details. (#2565)
• dials.index: Add indexing.method=pink_indexer for still images based on Gevorkov Y, Barty A, Brehm W, White TA, Tolstikova A, Wiedorn MO, et al. pinkIndexer – a universal indexer for pink-beam X-ray and electron diffraction snapshots. Acta Cryst A. 2020 Mar 1;76(2):121–31. (#2604)

Bugfixes

• dials.index: Correctly output imageset id when joint_indexing=False and max_lattices>1 (#2538)
• dials.index: Ensure the unindexed_reflections table in the indexer classes is updated during refinement. (#2545)
• Always use configured environment when bootstrapping CCTBX. This fixes building on MacOS Sonoma; for which the default libtbx configuration fails. (#2546)
• dials.cosym: Ensure correct reindexing of low symemtry data to the correct setting (#2548)
• dials.export: Unmerged MTZ output now uses gemmi, which fixes various issues (#1099, #1100, #2057, #2379, #2505) (#2549)
• dials.refine: Ensure online documentation is up to date. (#2551)
• dials.symmetry/dials.cosym: Avoid crashes when unable to find consistent symmetry during cell reduction routine. (#2552)
• dials.sequence_to_stills: Avoid creating an experiment for an image with no reflections on it. (#2554)
• dials.reciprocal_lattice_viewer: Fix toggling of ids on spotfinding output when viewing multiple files (#2566)
• dials.estimate_resolution: Fix bug causing fitting failures for isigma, misigma curves (#2568)
• dials.reciprocal_lattice_viewer: ensure the correct panel for the beam centre is recorded when loading models. (#2572)
• dials.reindex: When reindexing against a reference, use the best cell when determining lattice symmetry. (#2573)
• dials.cosym: Filter out datasets that can't be mapped through a consistent best cell, rather than exiting. (#2574)
• dials.export_bitmaps: Fix for use of imageset_index parameter on image sets that don't start at image numbers of 1. (#2591)
• Reorganize imports in export_mtz.py so that its gemmi-less objects can be used by other gemmi-less software (#2592)
• dials.image_viewer: Fix for stills experiments viewing crash introduced in #2556 (#2594)
• dials.refine: clarify a misleading error message when there are no reflections to refine. (#2607)

Misc

• #2408, #2547, #2550, #2556, #2557, #2563, #2570, #2576, #2578, #2586, #2590, #2599, #2600

DIALS 3.17.0 (2023-11-03)

Features

• Circular dependencies have been removed between DIALS and cctbx.xfel by using the new serialtbx. (#2404)
• dials.import: Enable use of an image range selection when importing a still sequence. (#2490)
• dials.estimate_resolution: Improved CC½ fitting by using a weighted tanh fit. (#2499)
• dials.find_spots and dials.integrate now accept exclude_images_multiple=N, which will be expanded to the appropriate exclude_images= definition that excludes every Nth image from the data set. This is intended for the situation where a scan of diffraction images is regularly interrupted by calibration images. (#2511)
• dials.ssx_integrate: Add mosaicity_max_limit= setting, to control what mosaicity limit is considered unphysically large. (#2518)
• dials.ssx_integrate: Add max_cell_volume_change_fraction= parameter to catch highly divergent cell refinements (#2521)

Bugfixes

• dials.cosym: Fix handling of refinement weights, for sparse data with a low number of common reflections. This improves the handling of serial still data. (#2503)
• dials.reflection_viewer: Fix a wsPython-related type error. (#2506)
• dials.sequence_to_stills: Now works with statically-refined experiments. (#2516)
• dials.sequence_to_stills: Generate identifiers for the split experiments. (#2517)
• dials.sequence_to_stills: No longer crashes when operating on a scan that does not start at image 1. (#2520)
• dials.symmetry: Print reindexed unit cell in log output. (#2523)
• dials.index: Correctly index multiple sweeps for multi-sweep indexed input. (#2534)
• dials.ssx_integrate: Fix bug in foreground mask calculation, affects datasets where wavelength is much bigger or smaller than 1 Å. (#2535)
• dials.index: Fix lattice counting when indexing with >1 sweep and >1 lattice (#2539)
• dials.cosym/dials.scale: Handle potential quasi_normalisation failures (for serial data). (#2540)
• Bootstrap --cmake: Fix configuration when CCP4 is already loaded. Previously, this could cause a failure as it tried to resolve the CCP4 copy of HDF5 in order to determine version. (#2544)

Deprecations and Removals

• dials.ssx_integrate: Remove duplicate named parameter output.shoeboxes= that has no effect (#2537)

Misc

• #2446, #2465, #2492, #2493, #2494, #2501, #2504, #2508, #2514, #2524, #2525

Dials 3.17 (2023-11-03)

Features

• Remove circular dependencies between DIALS and cctbx.xfel by using the new serialtbx. (#2404)
• dials.import: Enable use of an image range selection when importing a still sequence (#2490)
• dials.estimate_resolution: Improved cc1/2 fitting by using a weighted tanh fit (#2499)
• dials.find_spots and dials.integrate now accept a parameter exclude_images_multiple=N, which will be expanded to the appropriate exclude_images definition that excludes every Nth image from the data set. This is intended for the situation where a scan of diffraction images is regularly interrupted by calibration images. (#2511)
• dials.ssx_integrate: Allow setting of the mosaicity limit that is considered unphysically large, with mosaicity_max_limit option (#2518)

Bugfixes

• dials.cosym: Fix a bug affecting refinement weights for sparse data with a low number of common reflections. This improves the handling of serial still data. (#2503)
• dials.reflection_viewer: Fix wxpython-related type error. (#2506)
• The exclude_images_multiple parameter is adjusted to work correctly for scans that do not start with image number 1. (#2514)
• dials.sequence_to_stills: allow to work with statically-refined experiments. (#2516)
• dials.sequence_to_stills: Generate identifiers for the split experiments. (#2517)
• dials.sequence_to_stills: fix crash when operating on a scan that does not start at image 1. (#2520)
• dials.ssx_integrate: Add max_cell_volume_change_fraction parameter to catch highly divergent cell refinements (#2521)
• dials.symmetry: Print reindexed unit cell in log output (#2523)
• dials.index: Correctly index multiple sweeps for multi-sweep indexed input (#2534)
• dials.ssx_integrate: Fix bug in foreground mask calculation, affects datasets where wavelength is much bigger or smaller than 1. (#2535)
• dials.ssx_integrate: Remove duplicate named parameter output.shoeboxes that has no effect (#2537)
• dials.cosym/dials.scale: Handle potential quasi_normalisation failures (serial data) (#2540)
• Bootstrap --cmake: Fix configuration when CCP4 is already loaded. Previously, this could cause a failure as it tried to resolve the CCP4 copy of HDF5 in order to determine version. (#2544)

Misc

• #2446, #2465, #2492, #2493, #2494, #2501, #2504, #2508, #2524, #2525

DIALS 3.16.1 (2023-09-05)

Bugfixes

• dials.merge: Fix potential for crash with r_free_flags.extend=True, if there is no new flags to extend. (#2491)

DIALS 3.16.0 (2023-08-14)

Features

• dials.scale: Absorption correction is now switched off if all scattering angles are small (cos(2ϑ) ~ 1). (#2455)
• dials.export_bitmaps: Set resolution_rings.d_spacings to manually specify the size of resolution rings. (#2461)
• dials.index: Experiment number is now shown when indexing multiple experiments. (#2470)
• dials.export: You can now specifying a wavelength tolerance when exporting multi-wavelength data to mtz. (#2471)
• Support for building a CMake developer environment with bootstrap.py --cmake. This should be considered experimental, for now. (#2474)
• dials.merge: Added stereographic projections to the html report, if merging more than one experiment. (#2476)

Bugfixes

• dials.combine_experiments: No longer requires reflection data. (#2323)
• dials.model_background: Works again for data sets that do not start from image "1". (#2405)
• Updated DIALS to imageset slicing properly using 0-based indices. Previously, the slice had to be accessed with the original image indices. (#2411)
• dials.reindex: Fix crash reindexing against a reference file, for more than one input file. (#2439)
• dials.index: indexing.refinement_protocol.mode=None now applies to both the scans and stills indexers. (#2456)
• dials.index: Now fails with a useful error message when there are too few spots for indexing. (#2457)
• dials.ssx_integrate: Address issue with ellipsoid angular models by replacing with new, correct, r-dependent models. (#2463)
• dials.import: Now ensures manual geometry updates are only applied once to each model. This ensures invert_rotation_axis=True will only invert the rotation axis once. (#2469)
• dials.image_viewer: Fix crash when the detector is rotated more than 90° from the beam direction. (#2479)
• dials.reciprocal_lattice_viewer: Handle cases where a beam centre cannot be calculated, such as a detector with a high 2θ swing angle. (#2483)

Improved Documentation

• Update data files documentation to refer to MessagePack rather than pickle format. (#2444)

Deprecations and Removals

• dials.image_viewer: The non-functional "Show mask" checkbox has been removed from the Mask tool. (#2458)
• dials.image_viewer: The basis vector display has been removed. (#2484)

Misc

• #2390, #2433, #2434, #2435, #2437, #2438, #2445, #2462, #2466, #2472, #2482, #2489

DIALS 3.15.1 (2023-06-29)

Bugfixes

• dials.export_bitmaps: Fix the resolution_rings.fontsize= feature to work on Mac, and more reliably across platforms. (#2441)

DIALS 3.15.0 (2023-06-13)

Features

• dials.refine: Independent groups of experiments are now refined separately by default. This may not then produce identical output, but results are believed to be of equivalent quality. The previous behaviour can be selected by using separate_independent_sets=False. (#2336)
• Python 3.11 is now supported as a bootstrap target. (#2338)
• Support IOBS/SIGIOBS and FOBS/SIGFOBS columns when reading reference intensities from MTZ files. (#2386)
• dials.export_bitmaps: Optionally overlay ice rings. (#2393)
• export_bitmaps API code has been separated out of CLI script into the new dials.util.export_bitmaps module. (#2417)
• dials.reindex: Allow reindexing against a reference file (mtz, pdb, cif) from which intensities can be extracted/generated (#2420)
• dials.refine: new option auto_reduction.ignore=True allows avoiding the parameter auto-reduction procedure for special cases. (#2431)

Bugfixes

• dials.export_bitmaps: Correctly display the requested number of resolution rings. (#2397)
• More directly support cctbx bootstrap environments without fixed-version dependencies. (#2409)
• dials.scale: Fix crash in html plotting for scaling and filtering when R-pim/R-merge values contain 'None' values (#2410)
• dials.scale, dials.cosym: Use bulk solvent model when calculating intensities from a reference file. (#2414)
• dials.integrate: Fix multiprocessing on Windows with Python 3.11. (#2426)
• dials.ssx_integrate: Fix error in partiality calculation for ellipsoid integration algorithm. (#2429)
• dials.import: Don't print filename template for every image of an ImageSequence. (#2432)

Deprecations and Removals

• Moved dials::af::flex_table and associated tooling to dxtbx. Functions that are specific to reflection_table have been moved to reflection_table_suite.h. (#2113)
• dials.find_spots: Remove force_2d= option, which did not work as expected. (#2422)

Misc

• #2402, #2403, #2419, #2424, #2425, #2427, #2428

DIALS 3.14.2 (2023-05-16)

No significant changes.

DIALS 3.14.1 (2023-04-20)

Bugfixes

• dials.export_bitmaps: Correctly scale resolution rings with image binning= set. (#2392)
• Revert GTK-message-supression change that was causing reciprocal_lattice_viewer errors. (#2395)

DIALS 3.14.0 (2023-04-12)

Features

• dials.damage_analysis: Add dose_group_size= option to allow generation of plots for subsets of the data based on dose. (#1903)
• dials.import/dials.export: The image template is now reported in the log file, for rotation data. (#2278)
• Refinement tests now use publicly available data, though the dials-data package. (#2343)
• dials.merge: Now suggests a resolution limit, based on a fit to CC½. (#2351)
• dials.merge: Save merged half datasets to the MTZ file when additional_stats=True. (#2353)
• dials.export_bitmaps: Option to render resolution rings. (#2360)
• dials.rs_mapper: Now works with multi-panel detectors. (#2362)
• dials.ssx_integrate: Add support for multi-panel detectors, for the ellipsoid model. (#2368)
• dials.find_spots: Add the exclude_images= option. Previously this was only on dials.import. (#2374)
• dials.merge: Generate "FreeR_flag" column in output mtz, optionally copying from a reference mtz containing a FreeR_flag column. (#2378)

Bugfixes

• dials.refine: Does not call sys.exit when running as a library function. (#2325)
• dials.generate_mask: Fix a failure when the direct beam intersected the exact centre of a pixel. (#2329)
• Providing invalid PHIL parameters no longer causes a traceback. (#2331)
• dials.index: Handle possibility of Krivy-Gruber iteration limit exceeded in Niggli cell reduction. (#2332)
• CPU allocation limits will now be correctly inherited from the Slurm cluster scheduler. (#2335)
• dials.ssx_integrate: Fix cases where the background and centroid algorithms choices were not being used with ellipsoid-model integration. (#2337)
• dials.assign_experiment_identifiers now works with experiment lists without identifiers. (#2342)
• Test data files are no longer patched to correct their trusted range values, and usages of the deprecated DataBlock have been removed. (#2346)
• dials.ssx_integrate: catch rare ValueError in ellipsoid profile model refinement (#2349)
• dials.symmetry: Handle cases when an experiment has no scan object. (#2350)
• dials.ssx_index: Ensure file paths are split correctly on Windows. (#2358)
• Replace the deprecated imp library with importlib. (#2363)
• dials.rs_mapper: Fix bug causing the image data to be accessed with out-of-bounds indices. (#2364)
• dials.image_viewer: Fix error when using newer versions of wxPython. (#2365)
• DIALS viewers: Suppress terminal output saying "Gtk-Message". (#2366)
• Prevent occasional failures testing constrained geometry refinement. (#2367)
• Slicing a reflection_table does not lose the experiment identifiers map. (#2371)
• dials.integrate: Update exclude_images= syntax to match dials.import and dials.find_spots. The older form is now deprecated. (#2374)
• reflection_table.match_by_hkle: Fix failure on Windows with pandas 2.0.0. (#2383)

Improved Documentation

• Clarified coordinate system conventions in https://dials.github.io/documentation/conventions.html (#2373)

Misc

• #2251, #2324, #2344, #2345, #2352, #2356, #2357, #2361, #2369, #2372, #2381, #2385

DIALS 3.13.0 (2023-01-26)

Features

• dev.dials.napari_rlv: A reciprocal lattice viewer based on napari. This currently requires the napari module to be manually added into your DIALS installation. (#2229)
• dials.stills_process: Adds suppressed_logging= option, and minimal progress feedback. (#2263)
• dials.refine: Use an overall interval_width_degrees= parameter to set the default smoothness in scan-varying refinement for all models. This does not affect which models will be parameterised as scan-varying, which is controlled by their individual force_static= parameters. (#2268)
• Use Python 3.10 by default when running bootstrap. (#2272)
• dials.stills_process: Added reflection subsampling. If reflection_subsampling.enable=True, and an image fails to index, the reflections will be randomly subsampled, and indexing tried again. Reflections will be randomly subsampled in smaller amounts until a threshold is reached or the image successfully indexes. (#2275)
• dials.merge: Allow exclude_images= parameter, as in dials.scale. (#2280)
• dials.scale: Add ability to use a precalculated analytical correction as part of scaling models. (#2313)
• Add additional_stats option to generate R-split statistic for stills data in dials.merge and dials.scale. (#2314)

Bugfixes

• dials.integrate: Fail for negative profile-fitting intensity variance, not zero variance. (#2271)
• dials.import: Fix convert_stills_to_sequences= option for h5 data formats. (#2273)
• Slightly better support when handling empty reflection tables. (#2281)
• dials.scale: Catch rare crash when making summary stats table. (#2284)
• Fix wxPython 4.2.0 type error affecting RLV and geometry viewer. (#2287)
• Allow reflection_table.remove_on_experiment_identifiers() for an empty flex.reflection_table. (#2298)
• dials.cluster_unit_cell: Correctly report the number of lattices in each cluster, instead of the number of clusters. (#2300)
• dials.image_viewer: Fix error when using newer versions of wxPython. (#2306)
• dials.combine_experiments: Prevent default clustering.max_clusters=None from raising error when clustering. (#2311)
• dials.scale: Fix crash when scaling against reference, and some datasets were removed during scaling. (#2312)
• dials.integrate: Clearer error message when kapton absorption_correction.apply=True, but algorithm=None. (#2317)

Improved Documentation

• integration/corrections.h: Improve comments pertaining to the polarization correction. (#2274)

Misc

• #2266, #2267, #2269, #2282, #2289, #2290, #2293, #2301, #2303, #2307, #2308, #2309, #2310, #2319, #2321

DIALS 3.12.1 (2022-12-05)

No significant changes.

DIALS 3.12.0 (2022-10-31)

Features

• dials.stills_process: Add known_orientations= feature, useful for specifying previous processing results to use when reprocessing data, or for multiple detectors in an experiment. (#2110)
• dials.filter_reflections: Added remove_by_index= option. (#2201)
• dials.rs_mapper: Perform the calculation in parallel to improve speed. (#2238)
• dials.background: Add n_checkpoints= parameter, to run the analysis on evenly-spaced images. (#2240)
• dials.image_viewer: Change default brightness to 10. (#2254)
• The dials.ssx_index and dials.ssx_integrate programs are now considered stable, so have dropped the dev. prefix. (#2265)

Bugfixes

• dials.stills_process: Fix crashes for raster scans of stills. (#2128)
• dials.image_viewer: Fix intensity readout mouseover, and beamcenter half-pixel errors. (#2194)
• trusted_range is now defined throughout DIALS consistently as the inclusive range between the minimum and maximum trusted values - valid pixels are those less than or equal to the maximum trusted value and greater than or equal to the minimum trusted value. (#2195)
• dials.find_rotation_axis: Recover a plot that was broken by previous changes. (#2225)
• dials.scale: Restore consistent dataset id output numbering when a dataset is excluded. (#2246)
• dials.reciprocal_lattice_viewer: Set the maximum number of experiments to show in the selector toggles (default 15). Datasets with a number larger than this will not be individually selectable. (#2248)
• dials.ssx_integrate: Gracefully handle assertion error for particular bad data. (#2264)

Improved Documentation

• Removed references to pickle and json from PHIL config help strings. (#2208)

Deprecations and Removals

• The deprecated dials.algorithms.symmetry.cosym.target.Target.get_sym_ops() function has been removed. Please use the Target.sym_ops property instead. The nproc argument to the Target constructor has also been removed. The deprecated dials.util.mp.parallel_map function has been removed. The deprecated dials.command_line.dials_import.ImageImporter has been removed. Please use do_import instead. The command dials.rl_csv has been removed. Similar functionality is available with dials.export format=json. The command dials.find_shared_models has been removed. Use dials.show show_shared_modules=True ... instead. (#2212)

Misc

• #2230

DIALS 3.11.2 (2022-09-27)

Bugfixes

• dials.scale: Fix bug in intensity combination scoring for multi-sweep datasets, affecting midpoint test values. (#2199)

DIALS 3.11.1 (2022-09-02)

Bugfixes

• Revert default installation to Python 3.9, to avoid WXPython incompatibilities. (#2216)

DIALS 3.11.0 (2022-08-24)

Features

• dials.scale: Added the reference= option, with support for using a cif data file as a scaling reference. This replaces the old usages of target_mtz= and target_model=, which are now deprecated. (#2148)
• dials.algorithms.clustering.unit_cell API: Include the linkage matrix in the returned ClusteringResult object. (#2152)
• dials.cosym: Allow use of a reference dataset, to consistently index against. (#2154)
• dials.find_rotation_axis: Dramatically improve execution time by performing the search in parallel. (#2160)
• dials.image_viewer: Score tool now accepts keyboard entry. (#2162)
• dials.merge: Add merging statistics to output html report, and the PHIL option output.json= to write this this to json. (#2178)
• dials.symmetry: Add option to score systematic absences using fourier analysis. Thanks to Kevin Dalton for contributing this feature. (#2184)
• Unit cell clustering reports: Add plots of unit cell angle distribution. (#2197)
• Use Python 3.10 by default when running bootstrap. (#2206)
• dials.merge: include merging statistics in json output (#2207)

Bugfixes

• dials.find_rotation_axis: Consistently report rotation axis angle at start and end of the search. (#2145)
• dials.generate_mask: Make the log filename consistent with the program name. (#2147)
• dials.image_viewer: Automatically complete an active polygon mask when saving mask parameters. (#2150)
• dials.refine: Correct an error mapping constraint definitions to models. (#2155)
• dials.background: Check for empty experiments. (#2163)
• dials.estimate_gain: Now works on multiple experiments from a single image sequence. (#2164)
• dials.powder_calibrate: More general determination of background mask removal. (#2165)
• Fix DIALS version reporting for release builds. (#2166)
• Logging messages from dxtbx are now included in program output. (#2171)
• dev.dials.ssx_integrate: Handle potential assertion errors from FastMCD. (#2179)
• dials.symmetry: Ensure data for systematic absences check is in the correct setting for non-conventional minimum cells. (#2183)
• dials.scale: Fix bug in intensity combination scoring for multi-sweep datasets, affecting midpoint test values. (#2199)
• dev.dials.ssx_index: Fix crash when generating plotting data where rmsd values are almost all equivalent. (#2203)
• dev.dials.ssx_integrate: Fix divergent refinement bug when n_macro_cycles > 1. (#2204)

Deprecations and Removals

• dials.scale: The target_mtz= and target_model= options are deprecated. Please use reference= instead for both uses. (#2148)

Misc

• #2115, #2138, #2143, #2144, #2169, #2180, #2181, #2185, #2188

DIALS 3.10.3 (2022-08-02)

Bugfixes

• dials.image_viewer: Fix error after loading images with the "Load" button. (#2168)
• dials.merge: Fix crash for P-1 datasets. (#2175)
• dials.export format=mtz: Handle shared experiment models when converting to cambridge frame (#2182)

DIALS 3.10.2 (2022-07-13)

Bugfixes

• Fix DIALS version reporting for release builds. (#2166)

DIALS 3.10.1 (2022-07-12)

Bugfixes

• Refinement internals: Terminate Levenberg-Marquardt refinement if the objective is not expected to decrease. (#2135)
• dials.scale: Avoid edge case crashes when all reflections filtered out during reflection selection for a dataset in multi-dataset scaling (#2146)
• dials.cosym: Fix bug whereby the change of basis op was applied twice to the crystal model (#2151)
• dials.ssx_index: Fix potential numpy crash when trying to generate histograms from one datapoint (#2156)
• dials.image_viewer: Allow viewing still collections where some images have no reflections. (#2157)
• dev.dials.ssx_index: Fix cluster reporting for only one image indexed, fix crash when all images filtered out (#2159)

DIALS 3.10.0 (2022-06-09)

Features

• The DIALS code package now uses src/ layout. You will need to at least libtbx.refresh if on a development install. (#2077)
• dials.cosym: Enable skipping of unit cell clustering by setting unit_cell_clustering.threshold parameter to 0 or None. (#2058)
• dials.export: Add extra unmerged data categories for mmcif output, conforming to the latest mmcif_pdbx.dic. Make v5 the default (rather than v5_next). (#2078)
• dials.export: MTZ files are now exported with geometry in the Cambridge frame. (#2054)
• dials.index: Performance improvements for serial indexing (dials.stills-process, dev.dials.ssx_index). (#2035)
• dials.merge: New implementation of the French & Wilson (1978) algorithm for correction of negative intensities when estimating amplitudes. This implementation makes use of the standardized median as an M-estimator for the average intensity of resolution bins, which makes the procedure robust against the presence of very negative intensities. (#2100)
• dials.powder_calibrate: Add detector distance calibration. (#2075)
• dials.refine: New separate_images option performs outlier rejection on each image independently. (#2036)
• dials.refine: Parallelise outlier rejection to reduce overall run times. (#1427)
• dials.refine: Use sparse storage for scan-varying runs, reducing memory requirements and run times. (#2022)
• dials.scale: Allow use of a pdb model to calculate target intensities for scaling (phil option target_model). (#2053)
• dials.stills_process: Validate command-line arguments to prevent confusion when there is a typo. (#2106)
• dev.dials.ssx_index, dev.dials.ssx_integrate: Add option output.nuggets=, which can be used to specify a directory to which in-process results are stored. (#2114)
• Add a CMake build of DIALS. (#2096)

Bugfixes

• dials.cluster_unit_cell: Modify test so that it runs on Windows. (#2027)
• dials.cluster_unit_cells: Correctly handle cases with only one input crystal. (#2120)
• dials.combine_experiments: Exit with a helpful error message, if experiments have the same identifiers. (#2069)
• dials.export: Fix crash for exporting ssx data. (#2126)
• dials.find_bad_pixels: Output a mask file, as expected in the phil scope. Remove unimplemented png output option. (#2122)
• dials.image_viewer: Fix downstream SEGV involving wxPython. (#2134)
• dials.image_viewer: Fixed user setting for show_beam_center= being overridden. (#2103)
• dials.image_viewer: the n_iqr value for radial_profile thresholding is now correctly handled. (#2116)
• dials.import: When trying to import a missing file, say which file was missing. Print a warning if trying to import with a wildcard and no files were found. (#1863)
• dials.refine_bravais_settings: Ensure that the reported reindexing operators correctly map the input symmetry to the given Bravais settings, regardless of whether the input symmetry was a primitive or non-primitive setting. (#2105)
• dials.reindex: Fail with a helpful error message when attempting to reindex to a left-handed cell. (#1779)
• dials.scale: If scaling against a target, do targeted outlier rejection. (#2052)
• dials.scale: Respect user supplied high resolution limit in summary table. (#2118)
• dials.show: Avoid crash when an experiment does not have an imageset. (#2056)
• dev.dials.ssx_index: Fix reporting of results for h5 files, skip indexing of an image if fewer than min_spots= strong spots. (default 10). (#2055)
• dev.dials.ssx_index: Handle case where the input strong.refl file has no spots for some images. (#2039)
• dev.dials.ssx_integrate: Correctly handle input data containing multiple imagesets. (#2124)
• Handle reflection simulation case where test reflections could fail to generate. (#2094)

Improved Documentation

• dials.model_background: Add help message and test program. (#1109)
• Remove reference to nproc in tutorials where it is not needed. (#2030)
• Update DPF3 part 2 tutorial, to match current output. (#2030)
• Add dials.two_theta_refine to documentation (#2061)

Deprecations and Removals

• dials.find_shared_models has been retired. The command will now redirect users to dials.show, in combination with the show_shared_models=True option. This command stub will be removed in a future version. (#1070)
• dials.integrate: The unused background.algorithm=median has been removed. (#2066)
• The API interface dials.command_line.dials_import.ImageImporter is now deprecated. Please use ...dials_import.do_import instead. (#2080)

Misc

• #1973, #2037, #2038, #2041, #2043, #2047, #2051, #2062, #2065, #2070, #2071, #2073, #2074, #2079, #2081, #2082, #2083, #2086, #2087, #2088, #2089, #2091, #2092, #2093, #2095, #2099, #2101, #2102, #2104, #2112, #2119, #2131, #2133

DIALS 3.8.6 (2022-06-07)

• Resolve xia2 installation issues for downstream packaging.

DIALS 3.8.5 (2022-06-01)

Features

• dials.stills_process: validate command-line arguments to prevent confusion when there is a typo (#2106)

Bugfixes

• dials.show: Fix display of unknown vector columns. (#2048)
• dials.image_viewer: Fixed user setting for show_beam_center= being overridden. (#2103)

DIALS 3.9.2 (2022-05-09)

Bugfixes

• dials.show: Fix display of unknown vector columns. (#2048)

DIALS 3.8.4 (2022-04-01)

Bugfixes

• dials.scale: Fix crash when a dataset is filtered out during the scaling process (issue #2045). (#2045)

DIALS 3.9.1 (2022-03-31)

Bugfixes

• dials.scale: Fix crash when a dataset is filtered out during the scaling process (issue #2045). (#2045)

DIALS 3.9.0 (2022-03-14)

Features

• Bootstrap now allows creating a Python 3.10 environment. This should be considered experimental at this stage, and may fail because not all our dependencies have Python 3.10 support yet. (#1866)
• dials.export: Add SHELX .hkl file output. (#1925)
• dials.background now writes to a log file. (#1948)
• dials.cluster_unit_cell: Add option output.clusters=True/False to generate output files for each cluster generated by splitting the dendrogram at the given threshold. (#1950)
• Add reflection_table.match_by_hkle method, to match reflections between tables that have the same miller index and entering flags. (#1951)
• Add experimental dev.dials.ssx_integrate script for profile modelling and integration of SSX data, including ellipsoid profile modelling. (#1974)
• dials.reindex: Allow reindexing using multi-crystal reference data files. (#1977)
• add flex.reflection_table.concat method, to concatenate a list of reflection tables, including handling their experiment identifiers and ids (#1994)
• For data reduction programs, allow exclude_images option to take a single multi-sweep command, e.g. exclude_images=0:100:120,1:150:180 (#1996)
• Enable spot-finding threshold algorithms to use information about experimental models. (#2001)
• dials.find_spots: Added new spotfinder.threshold=radial_profile threshold algorithm. This calculates an average background in 2θ shells, and identifies peak pixels at a user-controllable level above the background. This simple method is particularly appropriate for cases with strong rotationally-symmetric background, such as electron diffraction images. An optional blurring function helps to suppress noise peaks and to join split spots. (#2009)
• dials.export: Add PETS 2 exporting with format=pets. This is used for processing electron diffraction data. (#2014)
• New tool - dials.powder_calibrate to help calibrate the geometry of an electron powder pattern. (#2016)
• Colours in plots: Some of our plots and figures were still using the Matplotlib colour map Jet. Matplotlib moved some time ago to using the perceptually uniform colour map Viridis, which was designed to be relatively colourblind-friendly, as its default. In this version of DIALS, we too have moved to use Viridis for our Plotly plots. With thanks to Ammaar Saeed (ammsa23) for this change. (#2026)

Bugfixes

• dials.refine could in some rare cases introducing an unphysical beam polarization vector. (#1939)
• dials.scale: Fixes to properly handle partiality of ssx data (#1965)
• This round includes modifications to enable Kapton absorption correction for higher angles of rotation of the Kapton tape. (#1968)
• ThreadPool and Socket resources were not properly closed in dials.find_spots_client. (#1976)
• Fix regression in installer python 2 compatibility. (#1990)
• Avoid using the uuid module, to avoid mpi errors on Python 3.8. (#2000)
• dials.refine: Avoid spike in memory usage while saving reflections. (#2024)

Improved Documentation

• Improved the "small molecule" tutorial by including the symmetry determination, scaling and export to e.g. SHELX format output. (#1900)
• Add a new multi-crystal tutorial, discussing the analysis of Br-lysozyme microcrystal data with dials.cosym and xia2.multiplex. (#1960)
• Update the betalactamase tutorial to use automatic scan-varying refinement. (#1971)
• Remove outdated developer tutorial. (#2003)
• Documentation font has changed to sans-serif. (#2010)

Deprecations and Removals

• Starting with this release DIALS requires a minimum Python version of 3.8. Bootstrap no longer allows the creation of Python 3.7 environments. (#1866)

Misc

• #1576, #1930, #1953, #1966, #1967, #1972, #1975, #1978, #1986, #1989, #1995, #1999, #2013, #2015, #2017, #2018, #2019, #2020, #2021, #2023

DIALS 3.8.3 (2022-02-22)

Bugfixes

• dials.cosym: Fix crash for edge case of a full dataset being excluded by the resolution filter (#1993)
• dials.two_theta_refine: Fix crash if running on scaled data with exclude_datasets=. (#2006)
• Fix downloads failing on MacOS with "426 Upgrade Required" (#2012)

DIALS 3.8.2 (2022-02-07)

No significant changes.

DIALS 3.8.1 (2022-01-25)

Features

• It is now possible to explicitly bootstrap all supported python versions. (#1988)

Bugfixes

• dials.refine: Fix crash from recording of parameter correlations in history.json. (#1923)
• Correct reporting of phi angle in scan-varying model plots. (#1929)
• dials.sequence_to_stills: correct error in the crystal model for scans that do not start from image 1. (#1933)
• dials.show: only show format class if meaningful (i.e. not Format or FormatMultiImage) (#1981)

Improved Documentation

• Update some Cosym and resolution-related PHIL descriptions. (#1969)

DIALS 3.8.0 (2022-01-11)

Features

• dials.indexed_as_integrated: manipulate an indexed reflection file to look as if it were summation integrated. This simply takes the spot intensities that have been indexed, assigns a resolution and sets the summation integrated flag. Allows symmetry analysis using intensities and scaling on indexed data for very rapid feedback data processing. (#1912)
• All command line programs now allow passing -h argument multiple times to increase verbosity. (#1920)
• Add Dockerfile to automatically build and push images on new releases. (#1936)
• Add experimental dev.dials.ssx_index script to index a block of ssx data. (#1955)
• Add alias dials.rlv for dials.reciprocal_lattice_viewer, and dials.rbs for dials.refine_bravais_settings. (#1959)

Bugfixes

• dials.import: Support arbitrary P1 known unit cells. (#1880)
• Utility fixes: Better handle cases of missing partiality data in reflection files. Treat missing resolution values as 0.0 instead of being empty. (#1911)
• dials.scale: Add missing "expids_and_image_ranges" information to the json output. This is required for some of the plots in dials.report output. (#1913)
• Copy docker entrypoint script with exec permissions. (#1940)
• Don't line-wrap command-line help messages (#1954)
• dials.check_indexing_symmetry: correctly handle d_max parameter if left at default value when d_min set. (#1957)
• Fixed bug that prevented the ability to plot absorption end of max and min due to Kapton (#1962)

Improved Documentation

• Updated MyD88 tutorial to make use of the new dials.find_rotation_axis command. (#1885)
• Document the dxtbx convention for representing the goniostat rotation operator \mathbf{R} on the conventions page. of the online documentation. (#1917)
• Update tutorial for DUI 2021.11.1. (#1938)

Deprecations and Removals

• The cosym nproc= and dials.util.parallel_map warnings have been made more visible. (#1909)

Misc

• #1907, #1908, #1910, #1928, #1947

DIALS 3.7.2 (2021-12-02)

Features

• dials.integrate: When determining available memory, take into account MemoryProvisioned from HTCondor machine ad if the _CONDOR_JOB_AD environment variable is set. nproc=auto: Take into account CpusProvisioned from HTCondor machine ad. (#1943)

Bugfixes

• Read _CONDOR_JOB_AD not _CONDOR_MACHINE_AD (#1945)

DIALS 3.7.1 (2021-11-17)

Bugfixes

• dials.export: No longer allow (erroneous) MTZ export for multiple experiments with multiple space groups. (#1915)
• dials.export: No longer fails for XDS_ASCII and SADABS export with intensity=auto. (#1926)
• dials.report: Fix broken json output option. Include more graphs in json output. (#1932)

DIALS 3.7.0 (2021-11-01)

Features

• Bootstrap support for MacOS M1 platforms. (#1841)
• New dials.find_rotation_axis program optimises the rotation axis from strong spot positions prior to indexing. (#1884)
• dials.import: Allow importing templates with no template characters. (#1840)
• dials.stills_process: Performance improvements in Kapton absorption correction and in rare cases of highly mosaic crystals. (#1846)
• dials.image_viewer Coordinates are now given in fast, slow order. (#1849)
• dials.image_viewer: Crystal basis vectors are now shown in the same colour as their predictions. (#1855)
• dials.image_viewer: Add the option to display the rotation axis (#1856)
• dials.image_viewer: Draw resolution rings for curved detectors. (#1899)
• dials.import: Unhandled files are now by default ignored. This means that e.g. *.log files alongside images will no longer prevent a successful import. Set ignore_unhandled=False to restore the previous behaviour. (#1881)
• dials.scale: Allow fixing of a particular correction with e.g. physical.correction.fix=absorption. (#1883)
• Installer now accepts a --raw-prefix option to use the target destination directly, instead of in a dials-X.Y subdirectory. (#1896)

Bugfixes

• dials.compute_delta_cchalf: Fix crash when only using passing dataset/group. (#1892)
• dials.find_bad_pixels: Pixel coordinates are now reported in row-major order, and mask value is now set to 16, which corresponds internally to "noisy pixel". (#1876)
• dials.find_rotation_axis: removed unused parameter optimise={True|False}. (#1898)
• dials.report: Don't show otherwise empty sections. (#1875)

Improved Documentation

• Improvements to 3DED tutorials. (#1850)
• SARS-CoV-2 main protease tutorial: process in C2 rather than I2 setting for consistency with published structures. (#1854)
• Removed outdated lysozyme nanocrystals tutorial. (#1877)
• Add an associated projects page to the website. (#1893)

Deprecations and Removals

• Bootstrap no longer allows creating Python 3.6 environments. (#1852)
• dials.util.mp: deprecate parallel_map() function and remove previously deprecated preserve_exception_message= parameters. (#1860)

Misc

• #1851, #1853, #1862, #1865, #1867, #1869, #1882, #1887, #1888, #1889, #1891, #1894, #1902

DIALS 3.6.2 (2021-09-21)

Bugfixes

• dials.reciprocal_lattice_viewer: In cases with multiple lattices, "Crystal Frame" now aligns all crystal frames, rather than just the first. Unindexed reflections are no longer shown in this mode. (#1868)

DIALS 3.6.1 (2021-09-06)

No significant changes.

DIALS 3.6.0 (2021-08-16)

This is the last release to support Python 3.6. Future releases will require a minimum of Python 3.7.

Features

• DIALS bootstrap now creates a Python 3.9 environment by default (#1735)
• New program: dials.reference_profile_viewer for viewing reference profiles dumped by dials.integrate when using the debug.reference.output=True option. (#1759)
• dials.combine_experiments: Unindexed reflections are now included in the combined output (#1760)
• dials.image_viewer: Image overlays are now accumulated over stacks of images (#1750)
• dials.image_viewer: Allow control of the basis vector scale from the settings window (#1780)
• dials.image_viewer: Better colour choice for text overlays. Labels will now be light grey on Black, or Dark grey on White. The previous settings were sometimes hard to read on narrow-contrast images. (#1781)
• dials.merge: Include DANO/SIGDANO columns in output .mtz when anomalous=True and truncate=True (#1809)
• dials.reciprocal_lattice_viewer: Show resolution on the "nearest point" label (#1770)
• dials.reciprocal_lattice_viewer now shows the path to the reflections in the title bar (#1771)
• dials.reciprocal_lattice_viewer: The default marker size now scaled automatically based on the data density (#1773)
• dials.scale: Always enable absorption correction if the absorption_level= parameter is set. Previously it was only enabled for sweeps >= 60° or if absorption_correction=True. (#1793)
• dials.scale: Allow a shared absorption correction between sweeps if using the physical model, with the option share.absorption=True. Extra absorption correction plots have also been added; and multiple sweeps are now aligned to the same reference frame. (#1811)
• API: ...scaling_library.scaled_data_as_miller_array now sets wavelength in the returned miller.array (#1808)
• reflection_table.match() now returns flex.size_t index arrays, instead of flex.int. (#1784)
• New bootstrap option: --conda to install with miniconda instead of micromamba. (#1730)

Bugfixes

• dials.combine_experiments: Correctly preserve mapping to images. This affects dials.image_viewer and dial.reciprocal_lattice_viewer. (#1093)
• dials.compute_delta_cchalf: Unwarranted precision in the output has been reduced (#1751)
• dials.find_spots: Fix counting of imagesets in histogram output (#1827)
• dials.image_viewer: Add buttons to clear unit cell and generic ring display (#1777)
• dials.image_viewer: Fix various minor behavioural bugs in the spot-finding and image type controls. (#1796)
• dials.import: Fail gracefully when # is missing from template. (#1840)
• dials.integrate: change default filename of debug reference profile to reference_profiles.pickle. (#1747)
• dials.integrate: Change default configuration so that unintegrated reflections are not retained. This helps reduce memory usage of downstream tools. Set output_unintegrated_reflections=True to restore the previous behaviour. (#1753)
• dials.integrate: ensure imageset_ids are always output. Affected use of image viewer, reciprocal lattice viewer on multi-sweep data. (#1762)
• dials.reciprocal_lattice_viewer: When starting with black_background=False, ensure the rotation axis and beam vector are displayed in black. (#1540)
• dials.reciprocal_lattice_viewer: More robust beam centre control that works for multiple panel detectors (#1842)
• dials.refine_bravais_settings: correctly report mI Bravais settings (#1825)
• dials.split_experiments: Update the imageset_id column in the output reflection files. (#1792)
• Don't fail bootstrap.py if a submodule is missing a reference (#1834)
• Correctly handle reflection imageset_id column in dials.scale, dials.cosym, and dials.symmetry. (#1763)

Improved Documentation

• dials.anvil_correction: Made a small improvement to the developer documentation. (#1788)
• Fix help string for best_monoclinic_beta= parameter (for dials.cosym, dials.refine_bravais_settings and dials.symmetry) (#1833)
• Added a new tutorial on 3DED/MicroED data processing. (#1837)
• Add a "Getting started" page to the documentation on the website. (#1844)
• Add a tutorial on processing small molecule 3DED data. (#1847)

Deprecations and Removals

• The previously deprecated dials.resolutionizer command has been removed. Please use dials.estimate_resolution instead. (#1330)
• The previously deprecated dials.refine parameter trim_scan_edges has been removed. Please use scan_margin=... instead. (#1374)
• The previously deprecated Spotfinder()() interface has been removed. Please use Spotfinder().find_spots() instead. (#1484)
• The previously deprecated dials.util.masking.MaskGenerator has been removed. Please use dials.util.masking.generate_mask instead. (#1569)
• The bootstrap option --mamba has become the default and will be removed in the future. (#1730)
• dials.anvil_correction: Drop compatibility support for SciPy < 1.4 (#1787)

Misc

• #1746, #1733, #1752, #1755, #1756, #1764, #1767, #1772, #1783, #1789, #1791, #1794, #1795, #1799, #1802, #1804, #1806, #1807, #1812, #1816, #1817, #1823, #1830, #1835, #1836, #1839

DIALS 3.5.4 (2021-07-27)

Bugfixes

• dials.stills_process: Fix case where imagesets and experiment filenames could potentially disagree (#1814)
• dials.scale: Fix incorrect output files, for targeted scaling with more than one target dataset. (#1815)
• dials.image_viewer: Fix opening datasets with load_models=False (#1818)

DIALS 3.5.3 (2021-07-12)

Bugfixes

• dials.image_viewer: Fix the basis_vector_scale= parameter. (#1769)

DIALS 3.5.2 (2021-06-28)

Bugfixes

• dials.image_viewer: Fix display of spotfinding intermediates (threshold, dispersion, etc) when viewing multiple still experiments (#1734)
• dials.image_viewer: Stacking images no longer gives incorrect results for multi-sweep data beyond the first sweep (#1758)

DIALS 3.5.1 (2021-06-14)

No significant changes.

DIALS 3.5.0 (2021-05-27)

Features

• dials.integrate: Avoid crash when data is too large to process in memory, by splitting into subsets (#1392)
• New bootstrap options: --mamba to install with micromamba, and --clean to remove installation caches immediately after completion. (#1676)
• dials.find_spots_server: Faster filtering of reflections by resolution (#1680)
• dials.scale: Add option error_model.grouping= to control refinement of either individual or grouped error models during scaling (#1684)
• dials.scale: Added physical.absorption_level=[low|medium|high] option for automatic setting of suitable absorption correction parameters. (#1688)
• dials.cosym: Significantly faster calculation of Rij matrix of pairwise correlation coefficients (#1693)
• dials.sort_reflections and dials.merge_reflection_lists are now available without a dev. prefix. (#1703)
• New command: dials.find_bad_pixels to identify pixels which are identified as signal in >= 50% of images (#1710)
• dials.image_viewer: Add selector to choose between a new default "image" and traditional "lab" coordinate frames. "image" frame attempts to align the fast/slow axes of the detector panels to screen x and y coordinates, so overall detector rotations will mostly be invisible. "lab" frame is the previous projection, where rotated detectors will appear rotated. dials.export_bitmaps: Gained this same projection= option. (#1716)
• dials.find_spots and dials.integrate: nproc= now works with N > 1 on Windows. (#1724)

Bugfixes

• Fix rare crash in symmetry calculations when no resolution limit could be calculated (#1641)
• dials.report: Add units of pixels / images to centroid difference histograms (#1677)
• dials.refine: Scan-varying refinement failed when trim_scan_to_observations=False was used. (#1686)
• dials.spot_counts_per_image: Show an explicit error if given data that isn't spotfinding output (i.e. unindexed reflections/experiments). (#1690)
• dials.integrate: Improved background model variance calculation for integrating detectors. (#1692)
• dials.stills_process: improve processing performance by preventing re-reading of image data (#1705)
• dials.background: Correctly identify signal pixels for integrating detectors, and respect pre-calculated masks. (#1726)
• dials.integrate: Fixed bug in memory-use calculation for multi-sweep integration runs (#1728)

Improved Documentation

• Remove remaining 'master' references in the documentation. (#1632)

Deprecations and Removals

• The previously deprecated dials.util.masking.MaskGenerator now prints a user warning. Please use dials.util.masking.generate_mask instead. (#1643)
• dials.cosym: Remove clustering code as this is no longer a necessary part of determination of symmetry or reindexing operations, and serves no useful purporse. (#1647)
• dials.cosym: nproc= parameter is deprecated. The algorithm is much faster on single cores. (#1693)
• The pytest option --runslow was retired. The tests that it triggered will now always run. (#1695)
• dev.dials.csv has been deprecated. Similar functionality is available with dials.export format=json. (#1708)
• dials.util.mp: The preserve_exception_message argument has been deprecated. (#1722)

Misc

• #1631, #1633, #1648, #1649, #1652, #1661, #1672, #1673, #1674, #1675, #1676, #1678, #1679, #1687, #1696, #1697, #1698, #1701, #1706, #1707, #1711, #1713, #1717, #1718, #1720

DIALS 3.4.3 (2021-04-20)

Bugfixes

• dials.scale: Fix crash when full-matrix minimisation is unsuccessful due to indeterminate normal equations. (#1653)
• dials.scale: Fix crash when no reflections remain after initial filtering. (#1654)
• dials.export: Fix error observed with format=mmcif for narrow sweeps with low symmetry (#1656)
• Fix image numbering inconsistency in ascii histogram of per-image spot counts (#1660)
• dials.find_spots_server: Significant performance improvement for HDF5 grid scans. (#1665)

DIALS 3.4.2 (2021-04-12)

Bugfixes

• Log messages from spot finding and integration no longer ignore logging level when using nproc > 1. This mainly affects usage of dials from outside contexts. (#1645)

DIALS 3.4.1 (2021-04-01)

Features

• dials.cosym: Significantly faster via improved computation of functional, gradients and curvatures (#1639)
• dials.integrate: Added parameter valid_foreground_threshold=, to require a minimum fraction of valid pixels before profile fitting is attempted (#1640)

Bugfixes

• dials.cosym: Cache cases where Rij is undefined, rather than recalculating each time. This can have significant performance benefits when handling large numbers of sparse data sets. (#1634)
• dials.cosym: Fix factor of 2 error when calculating target weights (#1635)
• dials.cosym: Fix broken engine=scipy option (#1636)
• dials.integrate: Reject reflections with a high number of invalid pixels, which were being integrated since 3.4.0. This restores better merging statistics, and prevents many reflections being incorrect profiled as zero-intensity. (#1640)

DIALS 3.4.0 (2021-03-15)

Features

• dials.integrate: Profile-fitting improvements; Profile fitting will now be attempted on reflections with masked pixels, and the number of reflections qualifying for profile-fitting on multi-panel detectors has dramatically increased. (#1297)
• dials.import: When using reference_models=, individual components of the model can be excluded with use_beam_reference=, use_gonio_reference= and use_detector_reference=. (#1371)
• flex.reflection_table.match can now match reflections with configurable distance and scaling between any 3-vector column in the reflection tables. The default is still "xyzobs.px.value". (#1398)
• dials.background: Add option output.plot= to save an image to disk, instead of displaying interactively. Image files can now also be used directly. (#1537)
• dials.import: The default tolerance.scan.oscillation= is increased to 3% of the image width, in order to accommodate electron diffraction datasets with poor rotation stages. (#1543)
• dials.background: Add support for multiple imagesets (#1554)
• dials.estimate_resolution: reject Wilson outliers to minimise effect of spurious observations from e.g. ice rings on the resulting resolution estimates (#1580)
• dials.cosym: Use numpy in place of flex for large parts of cosym analysis (#1581)
• dials.cosym: Add option to use scipy L-BFGS-B <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-lbfgsb.html> minimization engine (minimization.engine=scipy) (#1581)
• New masking parameter disable_parallax_correction=False. Set to True to speed up generation of resolution masks by disabling parallax correction (this is only likely to have significant effect when spotfinding is spread across many independent processes). (#1590)
• dials.image_viewer: New parameter basis_vector_scale= to adjust the length of the basis vector overlay (#1598)
• dials.merge: add option to set wavelength_tolerance for MAD datasets (#1609)
• dials.reciprocal_lattice_viewer: Added an option to label the reciprocal lattice point nearest the centre (#1614)
• dials.scale: An additional outlier rejection based on normalised intensities has been added (#1627)

Bugfixes

• dials.image_viewer: Fix various display issues relating to viewing still images (#1463)
• dials.background: Fix crash when writing output plot with bad display configuration (#1550)
• dials.scale: Fix issue of error model not always being carried through after the profile/summation intensity combination step. (#1566)
• Fail bootstrap step if the git checkout fails in a non-interactive or non-posix environment (#1572)
• Fixes working towards direct support of Windows builds:
  • Fix build errors by ensuring conda environment is correctly set up. (#1575)
  • Fix importing using paths with wildcards (#1583)
  • Fix dials.* commands crashing when unicode output is directed to a file (#1602)
  • Fix some type-related test failures (#1608)

Improved Documentation

• Describe how to fix gltbx build failures for development installations on non-RHEL distributions (#1561)
• Replace references to .pickle with reflections / .refl in docstrings (#1619)
• Add documentation for dials.filter_reflections to the website. (#1625)

Deprecations and Removals

• Remove previously deprecated use_trusted_range= parameter from masking configuration (#1156)
• The main development branch of dials was renamed from 'master' to 'main'. (#1546)
• dials.background: The plot= parameter to interactively display the background plot has been removed. Use output.plot= to save to file instead. (#1554)
• Remove *.o files from the DIALS installer package (#1564)
• dials.util.masking.MaskGenerator is deprecated in favour of dials.util.masking.generate_mask (#1569)

Misc

• #1530, #1531, #1532, #1534, #1535, #1536, #1542, #1567, #1570, #1571, #1588, #1593, #1597, #1599, #1600, #1601, #1603, #1604, #1613, #1620, #1621, #1624, #1626, #1630

DIALS 3.3.4 (2021-03-05)

Bugfixes

• dials.import: Selecting a subset of images with image_range= now works on stills (#1592)
• dials.search_beam_centre: Dramatically improved execution time for large data sets (#1612)
• dials.reindex: Write .refl file output in the default "MessagePack" format for better compatibility with downstream programs (#1616)
• dials.scale: Fix rare memory crash from infinite loop, that could occur with very bad quality datasets (#1622)

Improved Documentation

• dials.refine: More informative error message when reflections have weights of zero (#1584)

DIALS 3.3.3 (2021-02-15)

No changes to core DIALS in 3.3.3.

DIALS 3.3.2 (2021-02-01)

Bugfixes

• Remove unnecessary call to imageset.get_raw_data() while generating masks. This was causing performance issues when spotfinding. (#1449)
• dials.export: Allow data with either summation or profile fitted intensities to be exported. Previously, both were (erroneously) required to be present. (#1556)
• dials.scale: Fix crash if only summation intensities present and intensity_choice=combine (#1557)
• Fix unicode logging errors on Windows (#1565)

DIALS 3.3.1 (2021-01-18)

Features

• dials.index: More verbose debug logs when rejecting crystal models that are inconsistent with input symmetry (#1538)

Bugfixes

• dials.stills_process: Fix spotfinding error "Failed to remap experiment IDs" (#1180)
• Improved spotfinding performance for HDF5 when using a single processor. (#1539)

DIALS 3.3.0 (2021-01-04)

Features

• DIALS is now using GEMMI. (#1266)
• Upgrade h5py requirement to 3.1+ for SWMR-related functionality. (#1495)
• Added support for small integer types to DIALS flex arrays. (#1488)
• dials.estimate_resolution: Only use cc_half in default resolution analysis. (#1492)
• dials.export: Allow on-the-fly bzip2 or gzip compression for mmCIF output, because unmerged mmCIF reflection files are large. (#1480)
• dials.find_spots and dials.integrate both now have nproc=Auto by default, which uses the number of allowed/available cores detected. (#1441)
• dials.merge: Report <dF/s(dF)>, if anomalous=True. An html report is also generated to plot this statistic. (#1483)
• dials.scale: Apply a more realistic initial error model, or load the existing error model, if rescaling. (#1526)
• dials.stills_process: allow using different saturation cutoffs for indexing and integration. Useful for using saturated reflections for indexing while still rejecting them during integration. (#1473)

Bugfixes

• Internal: Logging metadata is now preserved when running spotfinding and integration across multiple processes. (#1484)
• Fix NXmx behaviour with h5py 3.1. (#1523)
• dials.cosym: Choose the cluster containing the most identity reindexing ops by default. Under some circumstances, particularly in the case of approximate pseudosymmetry, the previous behaviour could result in reindexing operators being chosen that weren't genuine indexing ambiguities, instead distorting the input unit cells. (#1514)
• dials.estimate_resolution: Handle very low multiplicity datasets without crashing, and better error handling. (#1494)
• dials.export,``dials.two_theta_refine``: Updates to mmcif output to conform to latest pdb dictionaries (v5). (#1528)
• dials.find_spots: fix crash when nproc=Auto. (#1019)
• dials.image_viewer: Fix crash on newer wxPython versions. (#1476)
• dials.index: Fix configuration error when there is more than one lattice search indexing method. (#1515)
• dials.merge: Fix incorrect output of SigF, N+, N- in merged.mtz. (#1522)
• dials.reciprocal_lattice_viewer: Fix error opening with wxPython 4.1+. (#1511)
• dials.scale: fix issues for some uses of multi-crystal rescaling if full_matrix=False. (#1479)

Improved Documentation

• Update information on how to care for an existing development environment, and remove outdated information. (#1472)
• Each of the available indexing strategies in dials.index now has some help text explaining how it works. You can view this help by calling dials.index -c -a1 -e1 and looking for method under indexing. (#1519)
• Include __init__ methods in autodoc generated library documentation. (#1520)
• dials.estimate_resolution: Improved documentation. (#1493)

Deprecations and Removals

• dials.algorithms.spot_finding.finder.SpotFinder: Use of __call__ to run spotfinding has been deprecated in favor of SpotFinder.find_spots(experiments). (#1484)

Misc

• #1469, #1481, #1484, #1487, #1491, #1496, #1497, #1498, #1499, #1500, #1501, #1514.

DIALS 3.2.3 (2020-12-07)

Bugfixes

• dials.slice_sequence: Fix crash using block_size= option (#1502)
• dials.scale: Fix broken exclude_images= option (#1509)

DIALS 3.2.2 (2020-11-23)

Bugfixes

• Fix case where dials.stills_process could swallow error messages
• dials.cosym: Fix non-determinism. Repeat runs will now give identical results. (#1490)
• Developers: Fix precommit installation failure on MacOS (#1489)

DIALS 3.2.1 (2020-11-09)

3.2 Branch releases will now use a fixed conda environment. This release is the first to use the same versions of all dependencies as 3.2.0.

Bugfixes

• dials.symmetry, dials.cosym and dials.two_theta_refine: Lowered default partiality_threshold from 0.99 to to 0.4. The previous default could occasionally result in too many reflections being rejected for particularly narrow wedges. (#1470)
• dials.stills_process Improve performance when using MPI by avoiding unnecessary log file writing (#1471)
• dials.scale: Fix scaling statistics output of r_anom data. (#1478)

DIALS 3.2.0 (2020-10-27)

Features

• DIALS development environments are now running Python 3.8 by default. (#1373)
• Add a scaled flag to the reflection table. Indicates which reflections are good after the scaling process. (#1377)
• Python warnings are now highlighted on the console log and written to log files (#1401)
• Exit error messages from commands will now be colourized (#1420)
• Change the way dials.integrate splits data into blocks, to reduce unnecessary data reads, increasing performance up to 35% in some cases (#1396)
• Add dials.util.mp.available_cores function (#1430)
• dials.refine: Trimming scans to observations for scan-varying refinement can now be turned off, using the parameter trim_scan_to_observations=False (#1374)
• dials.refine: Change default to separate_panels=False. This speeds up outlier rejection for multi-panel detectors. For metrology refinement this should be set to True (#1424)
• dials.merge: Add best_unit_cell option. If the best_unit_cell option is set in dials.scale, this will now propagate to the merged mtz output file. (#1444)
• DIALS bootstrap now allow creating a Python 3.9 environment (#1452)
• DIALS now uses pytype for limited static type checking. We hope that this will, over time, improve code quality. (#1364)
• dials.stills_process: Added process_percent= to restrict processing to a subset of data, sync reference geometry instead of overwriting it and handle composite spotfinding modes. (#1409)

Bugfixes

• dials.stills_process: Prevent memory usage getting too high by clearing the imageset cache during processing. (#1412)
• dials.find_spots_server: Return HTTP 500 instead of 200 when running fails (#1443)
• dials.find_spots_server: Fix multiprocessing-related crash on macOS with Python3.8 (#1447)
• dials.integrate: Fix failures when building with GCC 9 (#1456)
• dials.image_viewer: Fix deprecation warnings (#1462)
• dials.index: When using local index assignment, take into account phi in nearest neighbour analysis. This can significantly improve indexing rates in some cases with scans > 360° (#1459)
• dials.reindex: Show an error instead of crashing for bad reindex operations. (#1282)

Deprecations and Removals

• dials.refine: the parameter trim_scan_edges is renamed scan_margin and the former name is deprecated (#1374)
• The developer command dev.dials.show_test_failure_reasons was removed. (#1436)
• Remove clipper sources from new development installations (#1437)

Misc

• #1175, #1337, #1354, #1379, #1381, #1400, #1406, #1416, #1423, #1426, #1432, #1433, #1435, #1446, #1454, #1466, #1468

DIALS 3.1.4 (2020-10-12)

No changes to core DIALS in 3.1.4.

DIALS 3.1.3 (2020-09-28)

Bugfixes

• dials.integrate: fix integrator=3d_threaded crash if njobs > 1 (#1410)
• dials.integrate: Check for and show error message if shoebox data is missing (#1421)
• dials.refine: Avoid crash for experiments with zero reflections if the auto_reduction.action=remove option was active (#1417)

Improved Documentation

• dials.merge: improve help message by adding usage examples (#1413)
• dials.refine: More helpful error message when too few reflections (#1431)

DIALS 3.1.2 (2020-09-14)

Features

• dials.stills_process: Add parameter max_images= to limit the number of processed images

Bugfixes

• dials.integrate: fix crash when run with integrator=3d_threaded (#1404)
• dials.integrate: Minor performance improvements (#1399)
• dials.stills_process: MPI performance improvements for large datasets
• dials.stills_process: Fix error when using split logs

DIALS 3.1.1 (2020-09-01)

Bugfixes

• dials.scale: Prevent discarding of resolution limits in rare cases, which could cause incorrect symmetry determination, and worse results. (#1378)
• dials.cosym: filter out experiments with inconsistent unit cells (#1380)
• Internally slicing experiments now works if image range doesn't start at 1 (#1383)
• Restore missing I/sigma(I) resolution estimate log output (#1384)
• dials.image_viewer: "Save As" button now works, for single panels
• Fix developer libtbx.precommit installation error (#1375)

DIALS 3.1 (2020-08-17)

Features

• Supports Python 3.7 and 3.8. Python 3.6 remains the default. (#1236)
• Switch DIALS environment to use conda compilers. For development environments, a new dials script, located above the build directory, replaces the existing 'setpaths'-family of scripts. This means that all commands within the conda environment will now be available. (#1235)
• New command: dials.missing_reflections to identify connected regions of missing reflections in the asymmetric unit. (#1285)
• Improvements to image stacking in dials.image_viewer: - add pull-down selector for stacking mode - add modes for mean and max - add command-line selection for stacking mode - rename sum_images command-line option to stack_images (#1302)
• Reduce volume of output in dials.integrate; histograms and other less important information only shows in debug output. Pass the -vv option to restore the previous behaviour (#1319)
• dials.integrate: Experimental feature: Specifying output_unintegrated_reflections=False discards unintegrated data from output reflection file, for smaller output and faster post-processing (#1343)
• Rename dials.resolutionizer command to dials.estimate_resolution, and includes a html report. Writing png plot output is now turned off by default (passing plot=True will restore this behaviour). (#1330)
• dials.scale now separates anomalous pairs during error model analysis (#1332)
• dials.background: Add parameter corrected= to optionally use pedestal-and-gain corrected data (#1348)
• dials.combine_experiments: Add option output.max_reflections_per_experiment=, to reject experiments with too many reflections (#1369)

Bugfixes

• dials.image_viewer: Shoeboxes are now shown rotated with rotated detector panels. (#1189)
• dials.index: In multi-lattice indexing, ensure that reflections where refinement fails are flagged as unindexed. (#1350)
• dials.scale: Reflections excluded from scaling are no longer permanently excluded from any subsequent dials.scale jobs. (#1275)
• dials.scale: When using intensity_choice=combine (the default), don't exclude reflections that only have one of summed or profiled intensities available, but not both. (#1300)
• dials.split_experiments: Don't generate extra leading zeros in the output filename when not required e.g. output_09.expt -> output_9.expt (#1316)
• dials.plot_reflections: Fix invisible white spots on white background. (#1346)

Deprecations and Removals

• dials.find_spots: Deprecate spotfinder.filter.use_trusted_range= (#1156)
• setpaths.sh and related scripts in newly created DIALS development environments are made obsolete and will no longer work. (#1235)
• dials.show: Remove show_image_statistics= parameter. Use image_statistics.show_raw= for equivalent output (#1306)
• Log files will omit timings unless the relevant dials program was run with -v (#1313)

Misc

• #1184, #1216, #1288, #1312, #1320, #1322, #1325, #1328, #1352, #1365, #1366, #1370

DIALS 3.0.4 (2020-07-20)

• dials.scale: Allow usage of mode=image_group with filtering.method=deltacchalf when only providing a single data set (#1334)
• dials.import: When using a template and specifying an image_range, missing images outside of the range will not cause a failure (#1333)
• dials.stills_process: Show better error message in specific spotfinding failure case (#1180)

DIALS 3.0.3 (2020-07-06)

Features

• Developer tool: On posix systems, sending SIGUSR2 to DIALS commands will now print a stack trace (#1277)

Bugfixes

• HTML reports: Plot bin centres instead bin minimum for d_min line plots vs. resolution (#1323)
• dials.export: Fix inconsistency in mtz export when given a non-reference (e.g. I2 or primitive) setting (#1279)
• dials.refine_bravais_settings: Fix crash with large (>2gb) reflection tables and reduce memory use (#1274)
• dials.scale: Fix bug in outlier rejection code causing misidentification of outliers (with outlier_rejection=standard).
• dials.scale: Fix outlier rejection formula to avoid overconfidence in spuriously low values

DIALS 3.0.2 (2020-06-23)

Bugfixes

• Fix crash in scaling error model handling (#1243)

DIALS 3.0.1 (2020-06-11)

Features

• dials.reciprocal_lattice_viewer: Add an option to show lattice(s) in the crystal rather than laboratory frame. (#1259)
• Support for mtz project_name in export and scaling

Bugfixes

• dials.reciprocal_lattice_viewer: fix multiple experiment view for integrated data (#1284)

DIALS 3.0 (2020-05-22)

Features

• Show more useful output when crashing in C++ code (#659)
• dials.image_viewer: for the unit cell tool, rename parameters for consistency and add a new show_hkl option to filter displayed powder rings to select only those of interest. (#1192)
• In dials.integrate: changed the background box size multiplier to be a parameter (sigma_b_multiplier) - setting to small values significantly reduces memory requirements. (#1195)
• dials.image_viewer: add an overlaying showing pixels marked as strong by the spot-finding operations. That is, the pixels picked out by the "threshold" image. (#1200)
• dials.scale report file was renamed from scaling.html to dials.scale.html dials.symmetry report file was renamed from dials-symmetry.html to dials.symmetry.html (#1202)
• dials.report output file was renamed from dials-report.html to dials.report.html (#1206)
• dials.image_viewer: faster navigation between different image types. (#1213)
• Crystal model now has a new recalculated_unit_cell attribute. This allows it to store a post-refined unit cell (e.g. from dials.two_theta_refine) in addition to that from traditional geometry refinement (which was used for prediction). Downstream programs such as dials.scale and dials.export will now use the recalculated unit cell where appropriate. (#1214)
• New best_monoclinic_beta parameter for dials.refine_bravais_settings and dials.symmetry. Setting this to False will ensure that C2 is selected in preference to I2, where I2 would lead to a less oblique cell (i.e. smaller beta angle). (#1226)
• New scaling model, model=dose_decay, implementing a shared exponential decay component for multicrystal experiments (#1183)

Bugfixes

• Avoid empty "Unable to handle" messages on failed dials.import (#600)
• Functions from dials.export now raise exceptions on errors rather than exit. This improves their use elsewhere (such as in dials.scale). (#1205)
• Ensure dials.index chooses the C2 setting with the smallest beta angle (#1217)
• Fix propagation of best_unit_cell and application of resolution cutoffs in dials.scale and export_mtz. Add a new mtz.best_unit_cell parameter to dials.export (#1248)
• Make some of the DIALS tools furthest downstream (dials.scale, dials.symmetry, dials.merge and dials.report) more robust in the case of very few reflections. (#1263)

Misc

• #1221

DIALS 2.2 (2020-03-15)

Features

• dials.image_viewer: Add a choice between displaying the raw or the corrected image. (#634)
• Automatically choose between the simple and glm background determination algorithms, depending on whether the detector appears to be integrating or counting. (#706)
• Allow adjustment of font size for overlay text, such as Miller indices and resolution ring values. (#1074)
• Keep goniometer and scan objects in indexing of still data, if provided in input (#1076)
• Experimental: dials.image_viewer can be remotely controlled via a ZeroMQ endpoint with the zmq_endpoint PHIL parameter. Initially, the viewer can be commanded to load new images. This requires the (optional) ``pyzmq``package. (#1085)
• Programs now generate a unique identifier for each experiment created, and reflection tables are linked via the experiment_identifiers map (#1086)
• Introduce dials.anvil_correction to correct the absorption of the incident and diffracted X-ray beam by the diamond anvils in a pressure cell. Call dials.anvil_correction on the output of dials.integrate and then proceed to use post-integration tools as normal, just as though the sample had been measured in air. (#1090)
• Map of detector efficiency for photon counting detectors as a function of detector position added to report, based on the qe value applied at the end of integration. (#1108)
• Significantly reduce the amount of memory required to write .refl output files (#1115)
• Add maximum_trusted_value=N option to spot finding to temporarily allow override of trusted range, e.g. to find overloaded spots in spot finding. (#1157)
• array_family.flex interface has changed: background and centroid algorithms are set via public properties. Instead of flex.strategy use functools.partial with the same signature. as_miller_array() raises KeyError instead of Sorry. .extract_shoeboxes() lost its verbosity parameter, use log levels instead. (#1158)
• dials.stills_process now supports imagesets of length > 1 (e.g. grid scans) (#1174)

Bugfixes

• Fixed prediction on images numbered zero, so integrating works correctly. (#1128)
• Fix an issue (#1097) whereby aggregating small numbers of reflections into resolution bins could sometimes result in empty bins and consequent errors. (#1130)
• Ensure that restraints are ignored for parameterisations that are anyway fixed (#1142)
• Fix dials.search_beam_centre to ensure that the correct detector models are output when multiple detector models are present in the input. Fix dials.search_beam_centre n_macro_cycles option (previously it was starting from the original geometry every macro cycle). (#1145)
• dials.find_spots_server no longer slows down 3x when using resolution filters (#1170)

Misc

• #932, #1034, #1050, #1077

DIALS 2.1 (2019-12-12)

Features

• We now fully support Python 3 environments.
• MessagePack is now the default reflection table file format. Temporarily, the environment variable DIALS_USE_PICKLE can be used to revert to the previous pickle-based format, however this will be removed in a future version. (#986)
• new option for dials.show 'show_shared_models=True' displays which beam, crystal, and detector models are used across experiments (#996)
• Import still image sequence as N experiments dereferencing into one image set rather than one experiment. (#1014)
• Add reflection_table.get method for defaulted column access (#1031)

Bugfixes

• Don't use -2 to indicate masked pixels, except for DECTRIS detectors where this is to be expected. (#536)
• No longer show pixels that are above the trusted range upper bound as "saturated" on the "variance" image. (#846)
• Correctly account for scan-varying crystals while providing a scan range to dials.integrate (#962)
• Ensure that generated masks do not include pixels that are overloaded on a few images, but only pixels that are always outside the trusted range. (#978)
• Rewritten parameter auto-reduction code for dials.refine provides finer-grained fixing of individual parameters rather than whole parameterisations and correctly takes constrained parameters into account (#990)
• Fix output of predictions in dials.refine. A recently-introduced bug meant that the updated predictions weren't being copied to the output reflections file. (#991)
• Allow scan-varying refinement where either the crystal cell or orientation is fixed. (#999)
• Respect batch= option to dials.symmetry - can reduce time taken for finding the symmetry for large data sets. (#1000)
• Scan-varying refinement no longer fails when the scan is wider than the observed reflections (e.g. when the crystal has died). Instead, the scan is first trimmed to match the range of the diffraction. (#1025)
• If convert_sequences_to_stills then delete the goniometer and scan. (#1035)
• Correctly account for scan-varying crystals in dials.slice_sequence (#1040)
• Eliminate systematic absences before applying change of basis op to minimum cell in dials.symmetry. (#1064)

Improved Documentation

• Add "Extending DIALS" page to developer documentation (#893)

Deprecations and Removals

• The command dials.analyse_output was removed. Its replacement, dials.report, will give you more useful output. (#1009)

Misc

• #983, #1004

DIALS 2.0 (2019-10-23)

Features

• Support exporting multi-dataset and still experiments to XDS_ASCII (#637)
• Replace default spotfinder with improved dispersion algorithm (#758)
• dials.report now displays oscillation data with units and more significant figures (#896)
• A new program, dials.sequence_to_stills to create split a sequence into a separate still Experiment for every scan point in the sequence, splitting reflections as necessary. (#917)
• Moved dials.export format=best to dials.export_best as that one needed access to the format object, the rest do not, and having dials.export work in the general case seems like a better idea... (#921)
• Unified logging output for dials programs - logs are no longer split into .log and .debug.log. Use -v to get debug output. (#923)
• New command dials.resolutionizer (replaces xia2.resolutionizer). Add support for expt/refl in dials.resolutionizer. (#933)
• Changed the selection of reflections used for determination of the reflection profile parameters in integration. Now uses reflections which were previously used in refinement rather than all reflections, resulting in a speed improvement for large data sets and a negligible difference in the quality of the integrated results. (#942)
• dials.image_viewer now allows the choice between dispersion_extended (new default) and dispersion (old default) thresholding algorithms for investigating the effect of different spot-finding parameters. (#948)
• dials.rs_mapper now respects masked regions of images (including the trusted range mask). (#955)

Bugfixes

• Fix and reinstate normalisation option in dials.option (#919)

Misc

• #795, #862, #895, #915, #924