HIPAA and Medical Image Annotation: Why Offline Matters

Most image annotation tools are cloud-based: you upload images, annotate in the browser, download the results. For general computer vision work, that's fine. For medical AI teams working with X-rays, pathology slides, or endoscopy footage, it's a HIPAA risk.

The core issue: uploading a medical image to a third-party server is a disclosure of Protected Health Information (PHI) under HIPAA's Privacy Rule (45 CFR §164.502). That disclosure requires a signed Business Associate Agreement (BAA) with the vendor (45 CFR §164.504(e)). Annotation tool vendors typically offer BAAs only at enterprise pricing tiers, not on standard plans. And even with a BAA, you've expanded your attack surface: another server that touches your data.

The only way to eliminate that risk is to never upload the image in the first place.

LabelMe Has No Network Code

LabelMe is a desktop application. It reads image files from your local disk, lets you draw annotations, and writes JSON files back to disk. That's it.

There's no upload step. No telemetry. No license server pings. LabelMe contains no HTTP client code for sending images anywhere. You can run Wireshark while annotating for an hour and observe zero outbound connections related to your images.

This isn't a privacy mode you toggle on. The application simply doesn't have the capability to send your images out.

For HIPAA compliance, this distinction matters:

A cloud tool with a BAA still transmits your images to the vendor's servers. The BAA establishes liability; it doesn't prevent data from leaving your network.
An offline tool never transmits anything. There's nothing to audit because there's no data flow.

AI Annotation, Still Offline

LabelMe ships with SAM2, SAM3, and YOLO-World for one-click segmentation and text-prompt object detection, all running locally on your GPU or CPU. Model weights download once on first use, then everything runs fully offline. A medical team in an air-gapped hospital network can use AI-assisted segmentation without any data leaving the building.

For more on how offline AI annotation works, see Why Offline-First Annotation Matters for AI Teams.

Getting Started

Download LabelMe (Windows, macOS, Linux) or install via pip (see the installation docs). Annotations are saved as JSON files alongside each image, and the toolkit includes conversion scripts for COCO, VOC, and other formats.

For multi-annotator teams, annotations can be shared via institutional file shares or a HIPAA BAA-covered cloud storage provider. The annotation step itself stays local.

The compliance and cost case are the same

Cloud annotation platforms typically run $79–249/month per workspace. For a team spending that on annotating medical data, there's also the overhead of legal review for BAA negotiation, vendor security assessments, and ongoing compliance monitoring.

LabelMe Pro is $79, once. Your data never leaves your machine.

For medical AI teams that take HIPAA seriously, the architecture is the compliance strategy.