HIPAA and Medical Image Annotation: Why Offline Matters

Most image annotation tools are cloud-based: you upload images, annotate in the browser, download the results. For general computer vision work, that's fine. For medical AI teams working with X-rays, pathology slides, or endoscopy footage, it's a HIPAA risk.

The core issue: uploading a medical image to a third-party server is a disclosure of Protected Health Information (PHI) under HIPAA's Privacy Rule (45 CFR §164.502). That disclosure requires a signed Business Associate Agreement (BAA) with the vendor (45 CFR §164.504(e)). Annotation tool vendors typically offer BAAs only at enterprise pricing tiers, not on standard plans. And even with a BAA, you've expanded your attack surface: another server that touches your data.

The only way to eliminate that risk is to never upload the image in the first place.

Labelme Never Uploads Your Images#

Labelme is a desktop application. It reads image files from your local disk, lets you draw annotations, and writes JSON files back to disk. That's it.

There's no upload step and no license server. It connects to the internet to download AI models and, only if you agree to share usage data, to send a few app events that never contain images, annotations, or file paths. It never sends an image to a server. You can run Wireshark while annotating for an hour and observe zero outbound connections related to your images.

This isn't a privacy mode you toggle on. Labelme itself never uploads an image.

For HIPAA compliance, this distinction matters:

• A cloud tool with a BAA still transmits your images to the vendor's servers. The BAA establishes liability; it doesn't prevent data from leaving your network.
• An offline tool never transmits your images. There's nothing to audit because there's no data flow.

AI Annotation, Still Offline#

Labelme ships with SAM2, SAM3, and YOLO-World for one-click segmentation and text-prompt object detection, all running locally on your GPU or CPU. Model weights download once on first use, then everything runs fully offline. A medical team in an air-gapped hospital network can use AI-assisted segmentation without any data leaving the building.

AI Agents and Patient Images#

Labelme can connect to an AI agent such as ChatGPT or Claude. When the agent asks for the open image, Labelme hands it to the agent's app, which sends it to its model. For patient images, use the built-in models or an agent that runs its model on your computer.

For more on how offline AI annotation works, see Why Offline-First Annotation Matters for AI Teams.

Getting Started#

Download Labelme (Windows, macOS, Linux) or install via pip (see the installation docs). Annotations are saved as JSON files alongside each image, and the toolkit includes conversion scripts for YOLO, Pascal VOC, and other formats.

For multi-annotator teams, annotations can be shared via institutional file shares or a HIPAA BAA-covered cloud storage provider. The annotation step itself stays local.

The compliance and cost case are the same#

Cloud annotation platforms typically run $79–249/month per workspace. For a team spending that on annotating medical data, there's also the overhead of legal review for BAA negotiation, vendor security assessments, and ongoing compliance monitoring.

Labelme Pro is $79, once. Labelme never uploads your images.

For medical AI teams that take HIPAA seriously, the architecture is the compliance strategy.