0The image
Pick the slice you want to work on. Start with A. Each image keeps its own masks, so you can come back later and repeat the lab on another one.
Have a ground-truth mask for this image? Load it as a binary PNG (white = tumor, black = background, same shape as the image). It is then scored against every method in Exercise 4.
A and B: axial MRI of a glioblastoma by Hellerhoff, Wikimedia Commons, CC BY-SA 3.0, cropped and resized. C: generated for this lab.
What counts as “the tumor”? A glioblastoma has three parts: an enhancing rim that lights up after gadolinium contrast, a dark necrotic core of dead tissue inside the rim, and edema (swelling) spreading into the surrounding white matter. The BraTS benchmark calls rim plus core the tumor core, and that is what we outline in this lab: the bright rim and everything inside it, but not the edema.
Somebody chose that definition. Nothing in the image settles it for you.
1Draw it by hand
Hand-drawn outlines are still the reference standard. Radiotherapy targets are drawn by hand, and so is every mask in the datasets that segmentation networks are trained on. Drawing them is slow, though, and nobody draws the same outline twice.
- 1.1 Choose Outline and trace around the tumor in one stroke. When you let go, the inside fills in. Fix mistakes with Brush and Eraser.
- 1.2 Move the cursor slowly across the edge of the tumor and watch the magnifier. Is the edge a line, or a ramp?
- 1.3 When you are happy, press Draw it again. Your first mask is hidden. Outline the same tumor a second time, then press Compare attempts.
Questions
- Where along the boundary were you least sure, and why?
- A 3-D scan with 1 mm slices has about 150 axial slices, and a tumor this size spans roughly 40 of them. At your pace, how long would a full 3-D outline take?
- What Dice did you get between your two attempts? If you do not agree with yourself five minutes later, how well would you expect two radiologists to agree?
What you should notice
- Up close the edge is several pixels wide. Some of that is blur, and some is partial volume: a pixel on the boundary holds a bit of tumor and a bit of normal tissue, so its brightness falls in between. Wherever you put the line, you made a choice.
- Your two attempts mostly disagree in a thin band along the boundary. For a large tumor that band is a small share of the area, so Dice stays high. For a small lesion the same band would pull Dice down a lot.
- You used knowledge no algorithm on this page has: what a tumor looks like, where the ventricles are, and which bright spots are vessels.
2Otsu’s threshold
A threshold \(t\) splits the pixels into two classes: background (\(I \le t\)) and object (\(I > t\)). Otsu (1979) chooses the \(t\) that pushes the two classes furthest apart, measured by the between-class variance
\[ \sigma_B^2(t) = \omega_0(t)\,\omega_1(t)\,\bigl[\mu_0(t) - \mu_1(t)\bigr]^2 \]where \(\omega_0, \omega_1\) are the fractions of pixels in each class and \(\mu_0, \mu_1\) are their mean intensities. Otsu only ever looks at the histogram. It does not know where a pixel is or what its neighbors look like.
- 2.1 With Whole image selected, look at what Otsu calls object. Then drag the threshold in the histogram (Fig. 2b). Can any single \(t\) pick out only the tumor?
- 2.2 Drag a box around the tumor on the image. Now only the pixels inside the box go into the histogram. Where does \(t^*\) move, and why?
- 2.3 Switch on the clean-up steps one at a time and follow them in Fig. 2c. Click on the tumor to tell Keep one blob which blob you mean.
Fig. 2c. The binary mask after each step. White pixels are object. Faded steps are switched off.
Questions
- Otsu assumes there are two classes. How many kinds of tissue can you see in this image?
- Why does the center of the tumor come out as a hole? Which step fills it, and when would that step be wrong?
- Switch to image B (T2). What else is as bright as the tumor?
- Count the decisions you made: smoothing, box, threshold, opening, which blob, filling. Is this method automatic?
What you should notice
- On the whole image Otsu has to squeeze air, CSF, grey matter, white matter, fat, bone and tumor into two classes. On A it cuts between the brain and the bright scalp fat, so the object class is the ring of scalp plus the brightest parts of the rim. On B it collects CSF, tumor and scalp together. On C it simply separates air from head. The tumor never gets a class of its own.
- Inside a box the histogram holds only brain and tumor, so \(t^*\) moves to the gap between those two and the bright rim separates from the tissue around it. Anything in the box that is as bright as the rim still comes along, such as a vessel or the edge of a ventricle. You got rid of it by clicking on the right blob, so the spatial knowledge came from you, not from Otsu.
- The necrotic core is darker than \(t\), so it comes out as a hole. Fill holes repairs it, but only because you already know this tumor is solid.
3Active contour (snake)
A snake is a closed curve made of points \(v_1, \dots, v_n\). It slides over the image to lower an energy (Kass, Witkin and Terzopoulos, 1988):
\[ E = \sum_i \Bigl( \underbrace{\alpha\,\lvert v_{i+1}-v_i\rvert^2}_{\text{tension}} + \underbrace{\beta\,\lvert v_{i-1}-2v_i+v_{i+1}\rvert^2}_{\text{bending}} - \underbrace{w\,P(v_i)}_{\text{edges}} \Bigr) \]Here \(P = \lvert \nabla (G_\sigma * I) \rvert\) is the edge map: blur the image with a Gaussian of width \(\sigma\), then take the size of the gradient. Tension makes the curve contract like a rubber band. Bending keeps it smooth. The edge term pulls points onto strong edges. On top of that, a balloon force \(\kappa\) (Cohen, 1991) pushes every point along its normal: \(\kappa > 0\) inflates the curve and \(\kappa < 0\) deflates it. In each iteration every point takes a small step downhill.
- 3.1 Pick Draw a loop and draw a loop well outside the tumor. The curve starts moving as soon as you let go. Where does it stop?
- 3.2 Switch the background to Edge map, then turn on Force field. Arrows only appear near edges, because the snake cannot feel an edge more than a few σ away. Increase σ and look again.
- 3.3 Pick Click a seed and click in the middle of the tumor. The balloon inflates the curve until something stops it. Try κ = 0.3 and κ = 0.6, on image A and on image C. Where does the curve stop: on texture inside the tumor, at the inner edge of the bright rim, at its outer edge, or nowhere at all?
- 3.4 Set β to 0, then to 2. Then turn α up. Describe what each term does to the curve.
Region growing is a simpler alternative that works on regions instead of edges. Start from one seed pixel and keep adding neighbors whose intensity is within \(T\) of the region’s mean (Adams and Bischof, 1994). The animation adds one ring of neighbors at a time.
- 3.5 Click inside the tumor. Raise \(T\) slowly and watch for the moment the region leaks out.
- 3.6 Put a seed in the dark center, then one on the bright rim. Why are the results so different?
Questions
- Start the snake from two different places. Do you get the same answer?
- The skull and the ventricle walls are strong edges too. What keeps the snake off them, and when does that fail?
- Suppose the curve leaks out through a weak stretch of boundary. Which weight would you change, and what does that cost you elsewhere?
What you should notice
- The snake does not know what a tumor is. It stops at whatever edge is nearest and strong enough to hold off tension and balloon.
- Far from an edge the force field is zero. This is the capture range problem. A larger σ widens the range, but it also rounds off corners and can merge the two edges of a thin rim into one.
- Shrunk from outside, the loop usually wraps the tumor well, but it can snag on a nearby ventricle wall. Grown from a seed on image C, it stops at the inner edge of the rim and leaves the rim out. Both are local minima of the same energy, and which one you land in depends on where you started.
- On the real image A a seeded balloon often has no good setting at all. A weak balloon gets caught on texture inside the tumor. A strong one finds the weakest stretch of the rim, breaks through, and floods the brain until the skull stops it. That is why balloons are usually combined with region terms (as in Chan–Vese) or with a better edge map.
- Unlike a threshold, the result is always one smooth closed curve with no holes and no stray blobs. That built-in shape prior is the snake’s main strength.
- Region growing follows intensity, not edges. It leaks as soon as a neighbor of similar brightness touches the region, and a heterogeneous tumor leaves it full of holes.
4Compare the masks
These are the masks you made for the current image. Two numbers say how far apart two masks \(A\) and \(B\) are:
- Dice \(= \dfrac{2\,\lvert A \cap B\rvert}{\lvert A\rvert + \lvert B\rvert}\). It is 1 when the masks are identical and 0 when they do not overlap at all.
- HD95, the 95th-percentile Hausdorff distance. Take each boundary pixel of one mask, measure its distance to the nearest boundary pixel of the other, and report the 95th percentile of those distances, in pixels. It catches leaks and stray blobs that Dice barely registers.
Dice scores for the three methods
Dice between every pair
Questions
- Switch to the agreement map. Where do the methods agree, and where do they not?
- Image A has no ground truth. Which mask would you pick as the reference, and why is that choice uncomfortable?
- On image C, which method came closest to the truth? Is that the one you expected?
- Find a pair with a high Dice but a large HD95. What does the difference look like on the image?
What you should notice
- Most disagreement sits on the boundary. The rest shows up in a few typical places: the necrotic center (Otsu without hole filling), inner against outer rim (the snake), and leaks into neighboring tissue (region growing).
- Dice is forgiving for large objects. A small stray blob far away hardly changes it, but it sends HD95 up.
- For real patients the reference is a human drawing. Any method tuned or trained on such drawings, including deep networks trained on BraTS, inherits that person’s definition of the tumor and their variability. That takes you back to Exercise 1.
5Summary and reading
| By hand | Otsu threshold | Active contour | |
|---|---|---|---|
| What you supply | Every boundary pixel | Box, smoothing and clean-up steps (the threshold itself is automatic) | A starting curve and five weights: α, β, w, κ, σ |
| What it uses | Your knowledge of anatomy | The intensity histogram, nothing else | Edges, plus a preference for smooth curves |
| Typical failure | Slow; differs between attempts and between people | Takes everything that is equally bright; holes and stray blobs | Sticks to the nearest edge; leaks through weak edges; depends on the start |
| Shape of the result | Anything | Anything, often in pieces | One smooth closed curve |
| Same result twice? | No | Yes, given the same choices | Yes, given the same start and weights |
Going further
Gradient vector flow (Xu and Prince, 1998) spreads the edge forces outward so a snake can feel an edge from far away. Level sets and the Chan–Vese model (2001) use region statistics instead of edges and let the curve split and merge. Graph cuts find a global optimum instead of a local one. Today most tumor segmentation is done by convolutional networks such as the U-Net, trained on thousands of hand-drawn masks.
References
- N. Otsu. A threshold selection method from gray-level histograms. IEEE Trans. Systems, Man, and Cybernetics 9(1):62–66, 1979.
- M. Kass, A. Witkin, D. Terzopoulos. Snakes: active contour models. International Journal of Computer Vision 1(4):321–331, 1988.
- L. D. Cohen. On active contour models and balloons. CVGIP: Image Understanding 53(2):211–218, 1991.
- R. Adams, L. Bischof. Seeded region growing. IEEE Trans. Pattern Analysis and Machine Intelligence 16(6):641–647, 1994.
- C. Xu, J. L. Prince. Snakes, shapes, and gradient vector flow. IEEE Trans. Image Processing 7(3):359–369, 1998.
- T. F. Chan, L. A. Vese. Active contours without edges. IEEE Trans. Image Processing 10(2):266–277, 2001.
- O. Ronneberger, P. Fischer, T. Brox. U-Net: convolutional networks for biomedical image segmentation. MICCAI, LNCS 9351:234–241, 2015.
- B. H. Menze et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Trans. Medical Imaging 34(10):1993–2024, 2015.
- L. R. Dice. Measures of the amount of ecologic association between species. Ecology 26(3):297–302, 1945.