Models¶

pydantic model aind_watchdog_service.models.manifest_config.ManifestConfig[source]¶

Bases: BaseModel

Job configs for data transfer to VAST and executing a custom script

Show JSON schema

{
   "title": "ManifestConfig",
   "description": "Job configs for data transfer to VAST and executing a custom script",
   "type": "object",
   "properties": {
      "name": {
         "description": "Unique name for session data assets",
         "title": "Unique name",
         "type": "string"
      },
      "processor_full_name": {
         "description": "User who processed the data",
         "title": "Processor name",
         "type": "string"
      },
      "subject_id": {
         "description": "Subject ID",
         "title": "Subject ID",
         "type": "integer"
      },
      "acquisition_datetime": {
         "description": "Acquisition datetime",
         "format": "date-time",
         "title": "Acquisition datetime",
         "type": "string"
      },
      "schedule_time": {
         "anyOf": [
            {
               "format": "time",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Transfer time to schedule copy and upload. If None defaults to trigger the transfer immediately",
         "title": "APScheduler transfer time"
      },
      "force_cloud_sync": {
         "default": false,
         "description": "Overwrite data in AWS",
         "title": "Force cloud sync",
         "type": "boolean"
      },
      "transfer_endpoint": {
         "default": "http://aind-data-transfer-service/api/v1/submit_jobs",
         "description": "Transfer endpoint for data transfer",
         "title": "Transfer endpoint",
         "type": "string"
      },
      "platform": {
         "description": "Platform type",
         "enum": [
            "MRI",
            "exaSPIM",
            "confocal",
            "HCR",
            "behavior",
            "mesoSPIM",
            "MERFISH",
            "ISI",
            "HSFP",
            "ecephys",
            "motor-observatory",
            "single-plane-ophys",
            "multiplane-ophys",
            "SmartSPIM",
            "FIP",
            "SLAP2"
         ],
         "title": "Platform type",
         "type": "string"
      },
      "capsule_id": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Capsule ID of pipeline to run",
         "title": "Capsule"
      },
      "mount": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Mount point for pipeline run",
         "title": "Mount point"
      },
      "s3_bucket": {
         "$ref": "#/$defs/BucketType",
         "default": "private",
         "description": "s3 endpoint",
         "title": "S3 endpoint"
      },
      "project_name": {
         "description": "Project name",
         "title": "Project name",
         "type": "string"
      },
      "destination": {
         "description": "Remote directory on VAST where to copy the data to.",
         "examples": [
            "\\\\allen\\aind\\scratch\\test"
         ],
         "title": "Destination directory",
         "type": "string"
      },
      "modalities": {
         "additionalProperties": {
            "items": {
               "type": "string"
            },
            "type": "array"
         },
         "default": {},
         "description": "list of ModalityFile objects containing modality names and associated files or directories",
         "title": "modality files",
         "type": "object"
      },
      "schemas": {
         "default": [],
         "description": "Where schema files to be uploaded are saved",
         "items": {
            "type": "string"
         },
         "title": "Schema directory",
         "type": "array"
      },
      "script": {
         "additionalProperties": {
            "items": {
               "type": "string"
            },
            "type": "array"
         },
         "default": {},
         "description": "Set of commands to run in subprocess.",
         "title": "Commands",
         "type": "object"
      }
   },
   "$defs": {
      "BucketType": {
         "description": "Types of s3 bucket users can write to through service",
         "enum": [
            "private",
            "open",
            "scratch",
            "archive"
         ],
         "title": "BucketType",
         "type": "string"
      }
   },
   "additionalProperties": false,
   "required": [
      "name",
      "processor_full_name",
      "subject_id",
      "acquisition_datetime",
      "platform",
      "project_name",
      "destination"
   ]
}

Config:

extra: str = forbid

Fields:

acquisition_datetime (datetime.datetime)
capsule_id (str | None)
destination (str)
force_cloud_sync (bool)
modalities (Dict[Literal['MRI', 'SPIM', 'icephys', 'confocal', 'EMG', 'slap', 'fMOST', 'fib', 'behavior', 'ISI', 'merfish', 'ecephys', 'behavior-videos', 'pophys'], List[str]])
mount (str | None)
name (str)
platform (Literal['MRI', 'exaSPIM', 'confocal', 'HCR', 'behavior', 'mesoSPIM', 'MERFISH', 'ISI', 'HSFP', 'ecephys', 'motor-observatory', 'single-plane-ophys', 'multiplane-ophys', 'SmartSPIM', 'FIP', 'SLAP2'])
processor_full_name (str)
project_name (str)
s3_bucket (aind_data_transfer_models.s3_upload_configs.BucketType)
schedule_time (datetime.time | None)
schemas (List[str])
script (Dict[str, List[str]])
subject_id (int)
transfer_endpoint (str)

Validators:

normalize_modalities » modalities
normalize_platform » platform
normalized_scheduled_time » schedule_time
validate_capsule » all fields
validate_destination_path » destination
validate_modality_paths » modalities
validate_schema_paths » schemas

field acquisition_datetime: datetime [Required][source]¶

Acquisition datetime

Validated by:

validate_capsule

field capsule_id: str | None = None[source]¶

Capsule ID of pipeline to run

Validated by:

validate_capsule

field destination: str [Required][source]¶

Remote directory on VAST where to copy the data to.

Validated by:

validate_capsule
validate_destination_path

field force_cloud_sync: bool = False[source]¶

Overwrite data in AWS

Validated by:

validate_capsule

field modalities: List[str]] = {}[source]¶

list of ModalityFile objects containing modality names and associated files or directories

Validated by:

normalize_modalities
validate_capsule
validate_modality_paths

field mount: str | None = None[source]¶

Mount point for pipeline run

Validated by:

validate_capsule

field name: str [Required][source]¶

Unique name for session data assets

Validated by:

validate_capsule

field platform: Literal['MRI', 'exaSPIM', 'confocal', 'HCR', 'behavior', 'mesoSPIM', 'MERFISH', 'ISI', 'HSFP', 'ecephys', 'motor-observatory', 'single-plane-ophys', 'multiplane-ophys', 'SmartSPIM', 'FIP', 'SLAP2'] [Required][source]¶

Platform type

Validated by:

normalize_platform
validate_capsule

field processor_full_name: str [Required][source]¶

User who processed the data

Validated by:

validate_capsule

field project_name: str [Required][source]¶

Project name

Validated by:

validate_capsule

field s3_bucket: BucketType = BucketType.PRIVATE[source]¶

s3 endpoint

Validated by:

validate_capsule

field schedule_time: time | None = None[source]¶

Transfer time to schedule copy and upload. If None defaults to trigger the transfer immediately

Validated by:

normalized_scheduled_time
validate_capsule

field schemas: List[str] = [][source]¶

Where schema files to be uploaded are saved

Validated by:

validate_capsule
validate_schema_paths

field script: Dict[str, List[str]] = {}[source]¶

Set of commands to run in subprocess.

Validated by:

validate_capsule

field subject_id: int [Required][source]¶

Subject ID

Validated by:

validate_capsule

field transfer_endpoint: str = 'http://aind-data-transfer-service/api/v1/submit_jobs'[source]¶

Transfer endpoint for data transfer

Validated by:

validate_capsule

validator normalize_modalities » modalities[source]¶: Normalize modalities

validator normalize_platform » platform[source]¶: Normalize modalities

validator normalized_scheduled_time » schedule_time[source]¶: Normalize scheduled time

validator validate_capsule » all fields[source]¶: Validate capsule and mount

validator validate_destination_path » destination[source]¶

validator validate_modality_paths » modalities[source]¶

validator validate_schema_paths » schemas[source]¶

pydantic model aind_watchdog_service.models.watch_config.WatchConfig[source]¶

Bases: BaseModel

Configuration for rig

Show JSON schema

{
   "title": "WatchConfig",
   "description": "Configuration for rig",
   "type": "object",
   "properties": {
      "flag_dir": {
         "description": "Directory for watchdog to poll",
         "title": "Poll directory",
         "type": "string"
      },
      "manifest_complete": {
         "description": "Manifest directory for triggered data",
         "title": "Manifest complete directory",
         "type": "string"
      },
      "webhook_url": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Teams webhook url for user notification",
         "title": "Teams webhook url"
      },
      "misfire_grace_time_s": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": 10800,
         "description": "If the job scheduler is busy, wait this long before skipping a job. If None, allow the job to run no matter how late it is",
         "title": "Scheduler grace time"
      }
   },
   "required": [
      "flag_dir",
      "manifest_complete"
   ]
}

Fields:

flag_dir (str)
manifest_complete (str)
misfire_grace_time_s (int | None)
webhook_url (str | None)

field flag_dir: str [Required][source]¶: Directory for watchdog to poll

field manifest_complete: str [Required][source]¶: Manifest directory for triggered data

field misfire_grace_time_s: int | None = 10800[source]¶: If the job scheduler is busy, wait this long before skipping a job. If None, allow the job to run no matter how late it is

field webhook_url: str | None = None[source]¶: Teams webhook url for user notification