Recent work has demonstrated that automatic image-analysis techniques can ameliorate the difficulties of manual techniques (6-8). These efforts have all combined automatic processing with judicious manual interaction. Such semi-automatic methods typically involve the following basic analysis mode on a given 3D image:
For involved 3D medical applications, a combination of automated processing and manual user-interaction is ideal, since it exploits the user's complex cognitive skills and the computer's facility for processing large images. Unfortunately, the works of (6-7) are for specific applications and hence not generally applicable to other circumstances. Yoo et al. provide one specific interactive means for building up regions of interest from a presegmented image (8). Given the wide range of applications and imaging modalities available for 3D medical imaging, though, a more general semi-automatic environment is necessary. IMPROMPTU addresses the issue of providing a generic means for incorporating automatic image-analysis procedures into a 3D imaging application.
For a typical application, a suitable automatic image-analysis method, henceforth referred to as a process, is generally constructed by splitting the desired task into a sequence of subtasks. The subtasks are performed by specific image-processing functions, such as lowpass filtering, edge-detection, connected-components analysis, etc. For example, to detect long edge segments in a noisy image, one might construct the following process: la) lowpass filter to reduce noise (21); lb) apply a Kirsch edge operator to measure the intensity gradient (21); (c) perform nonmaxima suppression to reduce spurious high-gradient responses (22); (d) hysteresis threshold to find the strongest edge points (22); and (e) label connected components to save only the longest edge segments (23). Such processes are usually complex and hence take much trial-and-error experimentation (prototyping) to configure (9, 10). Vivid examples of such processes for 3D medical-imaging applications are discussed in the Applications section. Since different applications have different requirements, a process configured for a particular task is generally not applicable to other tasks [e.g., the LV-chamber extraction process of (e) cannot be used to solve the 3D coronary arterial-tree extraction problem of(ll)]. Thus, a convenient means is needed for constructing 3D image-analysis processes.
This paper describes a system for prototyping, testing, and executing generic 3D medical image-analysis processes. The system consists of a graphical user interface (GUI) tool that links to an extensive library of volumetric image-processing functions. The GUI system is called IMPROMPTU (IMage PROcessing Module for Prototyping, Testing, and Utilizing image-analysis processes) and the library is referred to as VIPLIB (Volumetric Image-Processing function LIBrary). IMPROMPTU has been designed to interact with VIDA, an existing 3D medical-imaging system for visualization and manual image-analysis (5). (IMPROMPTU can also be used in conjunction with ANALYZE, since both systems use the same image format. But the differing shared memory structures and Open Windows interfacing used by IMPROMPTU and VIDA relative to ANALYZE make this less convenient.) Thus, IMPROMPTU in conjunction with VIDA provides a general semi-automatic environment for 3D image analysis and visualization. Note that Khoros is a GUI-based system for configuring image-analysis processes (12). Khoros, while widely used and powerful, to date does not have automatic 3D-processing tools or tools for visualizing 3D data sets, does not interface readily to existing 3D medical image-analysis packages, is highly memory intensive (making it difficult to run on many platforms), and uses a non-standard interface.
The remainder of the paper describes the design and operation of IMPROMPTU and then gives examples of how to apply IMPROMPTU to several 3D medical-imaging applications.
Fig. 1. Composite window display for IMPROMPTU. The main IMPROMPTU panel shows a process fol LV chamber extraction (6).
Figure 1 gives a composite window layout of IMPROMPTU as it appears on a computer console. The system software is written in the C programming language and has been built within the standard X Windows environment using the XView and OpenWindows toolkits devised by Sun Microsystems. One interacts with the system using the keyboard and computer mouse. The basic mode of operation for IMPROMPTU is to load the necessary 3D images, set up VO, construct a process, and run the process. A process consists of a sequence of steps (image-processing functions). We emphasize that IMPROMPTU is not an image visualization package, even though some capability exists to preview leaded data (see Load Image Panel of Fig. 1). IMPROMPTU is designed to construct image analysis processes. Visual aids exist in IMPROMPTU to view information on constructed image-analysis processes and runs, as discussed in more detail below. A tool such as VIDA should be used to visualize extracted regions in analyzed images.
The remainder of this section discusses the design of IMPROMPTU. System operation is discussed in the subsequent section.
IMPROMPTU incorporates several powerful features for 3D biomedical image analysis. For this, the design of IMPROMPTU is structured around issues of modularity, flexibility, and user-friendly operation. Since IMPROMPTU can also be used for developing and prototyping new image-processing functions not included in VIPLIB, the design also addresses issues relevant to the function libraries linking to IMPROMPTU. Clearly, any function linked to IMPROMPTU can be distinctly different from other existing functions both in functionality and structure. It is therefore necessary to handle all functions in a uniform way in order to minimize the burden on algorithm developers. A new function can be easily added to IMPROMPTU by adhering to a few simple rules. The discussion below describes these rules and the internal structure of IMPROMPTU.
Fig. 2. Software system design for IMPROMPTU.
The overall design of the system software follows an object-oriented modular approach. The entire system is divided into three major subsystems that follow a hierarchical structure based on data and control flow. The three subsystems are: la) the main interface module (UI); lb) the function interface module (FIM); and (c) the function parameter module (FPM) (see Fig. 2). In keeping with the object-oriented design philosophy, the subsystems are designed so that they are functionally isolated from one another. For example, changes can be made to function parameter specifications in the FPM, without these changes having any impact on the execution of the UI or FIM modules. Such a design greatly facilitates adding new functions and changing existing functions. Also, the object-oriented philosophy results in a natural hierarchical flow of control and data information. Control and data information flow from the top-level UI module to the FIM, which in turn passes information to the FPM, which passes information to the individual functions. Specifics of the system modules are discussed below.
The UI is the top-level module of the system. This module provides the necessary mechanisms to: view, edit, load, save, and run processes; load, save, and subregion volumes; and create and execute scripts. The design of the UI provides a uniform means for algorithm developers to access available image-processing functions. This is accomplished by using data structures that are uniform and compact, yet flexible enough to handle different functions. One of these, the image data structure, contains information about image data sets, as highlighted below:
The UI module communicates with the FIM through pointers to the necessary images. For each image, one data structure pointer is passed to the FIM. No distinction is made at the UI level between the data structures used by individual functions. In effect, the UI module is completely oblivious to the specifics of the lower-level functions controlled by the FIM and FPM. All interfacing between the UI module and the actual functions is achieved through the FIM and FPM.
The FIM, the second-level module of the system, provides the necessary link between the UI module and the individual image-processing functions. The FIM gives the names of functions and their designated categories. This information is used to set up appropriate interface panels that present all functions and their categories as specified to the user-see example Functions Panel in Fig. 1. Even though the data structures pertaining to all functions is accessible, the FIM does not attempt to understand the individual functions, except for their names and categories. Once again, the uniformity of the data structures provides a means for isolating this module from the low-level functions controlled by the FPM and the high-level interface of the UI. User access to individual functions is accomplished by communicating with the FPM when necessary. Typically, when a user accesses a function by selecting it from the presented list in a category (e.g., in Fig. 1, the function "Binary Erosion" in the "Morphology" category has been selected), the FIM queries the FPM for information regarding that particular function. This information is made available to the FIM via a pointer to a parameter data structure, which is then used to set up interface panels for further user interaction. The parameter data structure contains all required information about a given function as high- lighted below:
Note, however, that no attempt is made at the FIM level to access the parameters directly. Any access to the parameters of specific functions is gained by appropriate subroutine calls to the corresponding FPM.
The FPM, the last module in the hierarchy, Is In actuality a set of separate modules that are specific to the image-processing functions. In fact, for every such function in the system, there is a corresponding FPM. At the beginning of execution (i.e., when the program starts up), the FPM reads and assembles all the parameters required by all functions into a common data structure. As mentioned above, the information in this data structure is used by the FIM to set up interface panels. The Parameters Panel shown in Fig. 1 illustrates the information assembled for the ''Binary Erosion" function. Since the setting up of interface panels and its components are handled by the FIM, the underlying X-windows programming required to construct these panels is rendered transparent to the algorithm developer. This allows someone with no X-windows programming experience to write and add their functions into IMPROMPTU with ease.
Some rules are enforced, however, in writing the "C" files for the functions to facilitate the FPM in assembling the data structures. To add a parameter to a function, developers have to use a specific subroutine call, whose arguments contain the name of the parameter, its default value, permitted range of values, etc. For example, to construct the parameter "number of iterations" illustrated in the Parameters Panel of Fig. 1, one would need the following statement:
This statement says that the parameter called "number of iterations" is an integer type with a default value of "1" and a range of "O" to ''20." The value "5" indicates that it is to be the fifth parameter in the panel, and the keyword NOVICE indicates that this parameter is a novice mode parameter for user-interaction. A novice parameter can be modified under both novice and expert modes of IMPROMPTU execution, whereas an expert mode parameter can only be modified under the expert mode of operation.
Since parameters of different types require different arguments, separate subroutines, with minor variations, are provided for each. Currently, IMPROMPTU supports the following types of parameters:
Integer--for typed-in integer values.
Float--for typed-in floating-point values.
Choice--for toggle- or menu-type parameters, where
the user can look at all the choices and select one.
String- -for character strings, such as file names.
Message-- for showing a message on a panel for annotation purposes.
When a user wishes to execute a constructed process, a mechanism is needed for executing all the function engines forming the process. This is accomplished by the FPM. The FPM provides a link to the actual subroutine that executes a given function. During a typical IMPROMPTU session, the user sets up a process containing one or more functions and also views and edits the parameters for each function in the process. When the process is executed tinteractively or otherwise), the UI module steps down the process, executing each function one at a time. Since the actual information corresponding to the functions is not accessible to the UI module, a request for execution is sent to the FIM. The FIM accesses the data structure corresponding to the function and extracts the name of the subroutine that actually contains the code for the function as specified by the FPM. A request is then generated by the FIM to the FPM to execute the function.
The design of IMPROMPTU also incorporates facilities for error handling and status reporting during execution. The types of messages that can occur are STATUS, FATAL, and WARNING and are classified according to their severity. Every message is presented along with the name of the module that generated it. STATUS messages are generated by the various IMPROMPTU modules to indicate their current execution status and successful completion. WARNING messages indicate errors during execution, that may allow the current module to complete, but can cause problems in other modules. FATAL errors are generated when the execution cannot continue further. The following is an excerpt from a sample IMPROMPTU run:
(STATUS) Main: Starting Background Processing
(STATUS) Load: Volume successfully leaded
(STATUS) Run: Running Step 1
(STATUS) Run: Running Step 2
(FATAL) Lowpass: filter dimension > subvolume size
(FATAL) Run: Execution Failed
(WARNING) Main: ERROR in processing
An image was leaded, and a two-step process was run on the leaded data. In the above example, the STATUS message generated by the "Main" module indicates the start of the execution in background mode. The STATUS message associated with ''Load'' mentions the loading of the volume data. The next two STATUS messages indicate the steps being run on the data. The FATAL message from the "Lowpass" function points out that the size of the filter selected is larger in some dimension than the leaded data. This error causes "Run" to stop as indicated by the next FATAL message. Such errors are propagated to the top-level UI module for appropriate action. In particu- lar, for the example above, the ''Main'' module reports an error in processing. By logging and analyzing all errors generated by IMPROMPTU, it is possible to debug the~ process under execution. During script-mode execution, the logged messages can be used to trace the complete process and locate any problems that may have affected the execution. In any case, the logged messages can be stored as text tiles for convenience and future reference.
The major task in running IMPROMPTU is the configuration of a process (Step 3 above). A process is configured as follows (see Fig. 1):
Note that, with IMPROMPTU, the user interacts with the function libraries, such as VIPLIB, through the Functions and Parameters windows. Facilities are provided under the File menu for saving, loading, and including processes. Users can save the current process by using the "Save Process" command. A saved process can be retrieved for further editing or execution by using the "Load Process" command. Frequently used portions of processes can be saved as subprocesses. The ",a href="include.gif">Include Process" command can be used for embedding such subprocesses inside a larger process. Using the Edit, File, and View facilities, users can create, edit, view, save, load and execute processes with relative ease. This provides a reconfigurable framework for image-analysis process development. Also, several temporary image slots are available for loading data and storing intermediate processing results. Fig. 1 depicts a typical process for 3D LV-chamber extraction.
Fig 1 highlights the function and parameter windows for Step [6] of this process. The Applications section gives further explanation of this example. Fig. 1 illustrates another feature of the system: the ability to set the user level. By appropriately setting "User level" on the Properties window, the user can operate IMPROMPTU in either an expert mode or a novice mode. In the expert mode, the user can set all available function parameters instead of relying on the default values. Of course, then, the user must understand in detail the operation of the functions. Less initiated users can run IMPROMPTU in the novice mode, whereby the more involved, but often less pertinent, parameters are dimmed out, such as the "Option:" parameter for Step [6] in Fig. 1.
IMPROMPTU has two process execution modes that can be invoked using the Run menu:
Interactive. IMPROMPTU runs as an interactive tool. This is useful for prototyping and testing fast- executing processes on small images. Processes are executed in the background to allow the user to continue to interact with the user interface.
Script (Batch). When a process has been prototyped, the user typically wishes to test the process's parameter sensitivity (try a large number of slightly differing processes on the same image) or to run a thoroughly tested process on a sequence of images. The script mode provides such a facility. The user can generate and save a UNIX scriptfile from within IMPROMPTU, which can then be executed in the batch mode. The scriptfile contains one UNIX command for each desired process run. Each command defines the input images, process names, subregion file name, etc.
Fig. 3. Process summary and command-file version of 3D LV-chamber extraction process. The command file can be edited as a text file and then run using IMPROMPTU.
Sophisticated UNIX users may also run IMPROMPTU in a non-interactive mode (without creat-ing scripts) by invoking the program with a specified set of command-line arguments. Finally, the user can also edit text-file versions of processes, which can then be used in either the interactive or batch mode. Text- files are created for all processes created and saved in IMPROMPTU. Such files can also be created by using any ascii-based text editor. Fig. 3 gives an example of a text file representing the LV-chamber extraction process (only a small portion of the Process Summary is shown for brevity).
Fig. 4. Summary of VIPLIB's available functions.
Image-Processing Function Library: VIPLIB A major strength of IMPROMPTU is the extensive collection of 1D, 2D, and 3D functions linked to it for constructing image-analysis processes. These functions, stored in the library VIPLIB, are highlighted in Fig. 4.
VIPLIB currently contains seven categories, as highlighted below:
Many of the functions, such as those in the Filter (17, 21, 23), Morphology (24), Topology (6, 21, 23), and Manipulation categories, involve standard image-processing operations, but can typically run on 1D, 2D, or 3D data. Others, such as those in the Segmentation and Measurement categories, are specially developed for multidimensional imaging applications.
The Turnkey category features functions that perform complex application-oriented tasks; each of these functions generally package into one convenient place a whole sequence of operations from the other categories. For example, the process depicted in Fig. 1 for 3D LV-chamber extraction is available as one turnkey function. Since many functions can perform several specialized operations, VIPLIB's breadth of capability is large.
As mentioned earlier, users can also link in any custom-made functions they construct, provided that the functions conform to the image and parameter formats assumed by IMPROMPTU. The system's modular software design allows the interface to run independent of the linked functions. Hence, from a software- design standpoint, the GUI interface is oblivious to the functions available.
Fig. 5. Manually edited slice of a 3D CT cardiac image. The deleted dark area gives a separation between the LV chamber and the LA chamber and aorta.
We first consider the problem of extracting the LV chamber from a 3D DSR cardiac image, which was previously discussed and validated in (6, 14). In image-analysis terms, the basic goal is to extract a large, bright, solid region that does not have thin attachments. The input image is corrupted by noise and reconstruction artifacts and the desired LV chamber, while globally bright, varies in intensity across its extent and has weak region borders in certain portions of its structure. The first steP in extracting the LV chamber is to have the user interactively insert several cuts that aid in delineating the atrioventricular valve and aortic valve regions. This can be done using interactive slice tracing and only takes a few minutes of operator interaction for a typical 3D image (see Fig. 5).
The atrial/ventricular boundary is obscured by partial volume effects due to the thin valve leaflets and bright contrast agents in both chambers. Also, at essentially all times, one of the valves will be open. Hence, the manually introduced divisions greatly assist IMPROMPTU's subsequent automatic processing to extract the LV chamber. Fig. 1 summarizes the automatic process, constructed in IMPROMPTU, for 3D LV chamber extraction. The basic sequence of operations is given below (the italicized terms give the VIPLIB function names):
Fig. 6. Original 3D CT cardiac image lA) and extracted LV chamber (B).
Explicit details on this procedure can be found in (6). Figure 6 depicts a 3D shaded-surface display of a typical processed image, taking advantage of IMPROMPTU running within the VIDA visualization environment. This procedure takes considerably less operator interaction than manual slice editing and gives more reproducible, accurate results. Since this process has been extensively validated (14), it is incorporated as a Turnkey function in VIPLIB.
Fig. 7. 3D myocardium extraction using cue-based relaxation labeling (15).
Figure 7 gives an example of myocardium extraction from a 3D cardiac image. This example uses interactive relaxation labeling, which is discussed in detail in (15). To use this procedure, one first creates cues interactively via region painting or slice editing in a tool such as VIDA's 3D manual segmentation module (5) or in INTERSEG, a tool being designed for interactive semi-automatic 3D image segmentation (15). The cues give information on the properties of the regions of interest. Within IMPROMPTU, one can then use the create cues and relaxation labeling functions to perform automatic 3D image segmentation.
References (11, 16, 17) document other 3D car- diac Imaglng applications for the problems of angio- gram analysis, image segmentation, and image en- hancement. The function engines built for these applications have been incorporated into VIPLIB.
Fig. 8. Manually edited MR image of the upper airway. The general area containing the upper airway has been ioosely circled.
Fig. 9. Original 3D MR image lA) and the extracted upper airway (B). Automatic extraction performed on manually edited image (Fig. 8) using IMPROMPTU.
As a final example, Figs. 8 and 9 depict results for a semi-automatic procedure for 3D upper-airway extraction from a 3D MRI data set (18). The basic procedure involves several stages. First, the user manually isolates the general area encompassing the airway in a particular image, again using a manual segmentation tool such as slice editing (see Fig. 8). One then runs the following automatic process, built within IMPROMPTU, on the partially edited image: la) sigma Jilter, to enhance regional borders and reduce noise (17); lb) region growing, to extract the area encompassing the upper airway cavity (16); (c) cavity deletion, to generate a solid structure (6). At this point, the user again employs a manual slice editing to isolate the general area belonging to the airway (much data has already been excluded before this operation). One then uses IMPROMPTU again to isolate the largest connected region (using the function 3D connected components).
The problem of how to analyze and visualize data has been a major bottleneck in the widespread use of 3D imaging. Any results garnered from a 3D-imaging protocol must be believable and reproducible; also, to justify resorting to 3D Imagmg, the results must be unattainable from conventional 2D slice imaging. To achieve this level of performance, a typical scenario for handling a 3D medical imaging application would be to use IMPROMPTU in conjunction with a powerful 3D visualization package (3-5). This provides a combination of sophisticated automatic processing and manual expert interaction. The examples given in the Applications section demonstrate the efficacy of this strategy. For some "highly-controlled" specific 3D applications, though, we believe that purely automatic schemes should be possible. But eliminating manual interaction should not be achieved at the cost of losing accuracy or reproducibility.
Any results garnered from a 3D-imaging protocol must be believable and reproducible. Also, to justify resorting to 3D imaging, the results must unattainable from conventional 2D slice imaging. To achieve this level of performance, a typical scenario for handling a 3D medical imaging application would be to use IMPROMPTU in conjunction with a powerful 3D visualization package. This would provide a combination of sophisticated automatic processing and manual expert interaction. To this end, IMPROMPTU has been designed so that it can interact with VIDA's shared memory structure. This gives a strong environment for 3D image-analysis, consisting of tools for visualization, manual interaction, and automatic processing. But IMPROMPTU can readily be configured to run in other customized environments that use standard Xwindows/Open- Look protocols. 3D imaging applications for left- ventricular chamber extraction, myocardium extraction, and upper-airway analysis are given to demonstrate the functionality of IMPROMPTU.
Acknowledgments-This work was supported in part by NIH grants
CA-53607, HL-42672, and HL-43286. The 3D cardiac images were
provided by Drs. Erik Ritman and Namsik Chung of The Mayo
Clinic, Rochester, MN. The authors thank Doug Castor, Mike Han-
sen, Brian Ledell, Eric Ojard, Joe Reinhardt, and Werner Sharp for
their efforts in contributing to VIPLIB. Portions of this manuscript
appeared in the conference publication (19).
