data_transfer.aind_watchdog

WatchdogSettings

Bases: ServiceSettings

Settings for the WatchdogDataTransferService.

Attributes:

Name	Type	Description
destination	Path	The destination path for the data transfer.
schedule_time	Optional[time]	The time to schedule the data transfer.
project_name	str	The name of the project.
platform	Platform	The platform of the project.
capsule_id	Optional[str]	The capsule ID for the data transfer.
script	Optional[Dict[str, List[str]]]	A dictionary of scripts to run.
s3_bucket	BucketType	The S3 bucket to transfer the data to.
mount	Optional[str]	The mount point for the data transfer.
force_cloud_sync	bool	Whether to force a cloud sync.
transfer_endpoint	str	The endpoint for the data transfer service.
delete_modalities_source_after_success	bool	Whether to delete the source data after a successful transfer.
extra_identifying_info	Optional[dict]	Extra identifying information for the data transfer.
upload_tasks	Optional[Any]	Upload job configurations. Use the placeholder "{{ destination }}" to later reference the destination path.
job_config	str	Job configuration name.

settings_customise_sources classmethod

settings_customise_sources(
    settings_cls: Type[BaseSettings],
    init_settings: PydanticBaseSettingsSource,
    env_settings: PydanticBaseSettingsSource,
    dotenv_settings: PydanticBaseSettingsSource,
    file_secret_settings: PydanticBaseSettingsSource,
) -> Tuple[PydanticBaseSettingsSource, ...]

Customizes the settings sources to include the safe YAML settings source.

Parameters:

Name	Type	Description	Default
settings_cls	Type[BaseSettings]	The settings class.	required
init_settings	PydanticBaseSettingsSource	The initial settings source.	required
env_settings	PydanticBaseSettingsSource	The environment settings source.	required
dotenv_settings	PydanticBaseSettingsSource	The dotenv settings source.	required
file_secret_settings	PydanticBaseSettingsSource	The file secret settings source.	required

Returns:

Type	Description
Tuple[PydanticBaseSettingsSource, ...]	A tuple of settings sources.

Source code in src/clabe/services.py

@classmethod
def settings_customise_sources(
    cls,
    settings_cls: t.Type[ps.BaseSettings],
    init_settings: ps.PydanticBaseSettingsSource,
    env_settings: ps.PydanticBaseSettingsSource,
    dotenv_settings: ps.PydanticBaseSettingsSource,
    file_secret_settings: ps.PydanticBaseSettingsSource,
) -> t.Tuple[ps.PydanticBaseSettingsSource, ...]:
    """
    Customizes the settings sources to include the safe YAML settings source.

    Args:
        settings_cls: The settings class.
        init_settings: The initial settings source.
        env_settings: The environment settings source.
        dotenv_settings: The dotenv settings source.
        file_secret_settings: The file secret settings source.

    Returns:
        A tuple of settings sources.
    """
    return (
        init_settings,
        _SafeYamlSettingsSource(settings_cls),
        env_settings,
        dotenv_settings,
        file_secret_settings,
    )

WatchdogDataTransferService

WatchdogDataTransferService(
    source: PathLike,
    settings: WatchdogSettings,
    *,
    validate: bool = True,
    session_name: Optional[str] = None,
    ui_helper: Optional[UiHelper] = None,
    email_from_experimenter_builder: Optional[
        Callable[[str], str]
    ] = lambda user_name: f"{user_name}@alleninstitute.org",
)

Bases: DataTransfer[WatchdogSettings], Generic[TSessionMapper]

A data transfer service that uses the aind-watchdog-service to monitor and transfer data based on manifest configurations.

This service integrates with the AIND data transfer infrastructure to automatically monitor directories for new data and transfer it to specified destinations with proper metadata handling and validation.

Attributes:

Name	Type	Description
_source	PathLike	Source directory to monitor
_settings	WatchdogSettings	Service settings containing destination and configuration
_aind_session_data_mapper	Optional[_TSessionMapper]	Mapper for session data
_ui_helper	UiHelper	UI helper for user prompts

Example

# Basic watchdog service setup:
settings = WatchdogSettings(
    destination="//server/data/session_001",
    project_name="my_project"
)
service = WatchdogDataTransferService(
    source="C:/data/session_001",
    settings=settings
)

# Full configuration with session mapper:
settings = WatchdogSettings(
    destination="//server/data/session_001",
    project_name="behavior_study",
    schedule_time=datetime.time(hour=22, minute=30),
    platform=Platform.BEHAVIOR,
    force_cloud_sync=True
)
session_mapper = MySessionMapper(session_data)
service = WatchdogDataTransferService(
    source="C:/data/session_001",
    settings=settings
)
service = service.with_aind_session_data_mapper(session_mapper)
if service.validate():
    service.transfer()

Initializes the WatchdogDataTransferService.

Parameters:

Name	Type	Description	Default
source	PathLike	The source directory or file to monitor	required
settings	WatchdogSettings	WatchdogSettings containing destination and configuration options	required
validate	bool	Whether to validate the project name	True
session_name	Optional[str]	Name of the session	None
ui_helper	Optional[UiHelper]	UI helper for user prompts	None

Example

# Basic initialization:
settings = WatchdogSettings(
    destination="//server/archive/session_001",
    project_name="behavior_project"
)
service = WatchdogDataTransferService(
    source="C:/data/session_001",
    settings=settings
)

# Advanced configuration:
settings = WatchdogSettings(
    destination="//server/archive/session_001",
    project_name="behavior_project",
    schedule_time=datetime.time(hour=23),
    platform=Platform.BEHAVIOR,
    force_cloud_sync=True,
    delete_modalities_source_after_success=True,
    extra_identifying_info={"experiment_type": "foraging"}
)
service = WatchdogDataTransferService(
    source="C:/data/session_001",
    settings=settings
)

Source code in src/clabe/data_transfer/aind_watchdog.py

def __init__(
    self,
    source: PathLike,
    settings: WatchdogSettings,
    *,
    validate: bool = True,
    session_name: Optional[str] = None,
    ui_helper: Optional[ui.UiHelper] = None,
    email_from_experimenter_builder: Optional[
        Callable[[str], str]
    ] = lambda user_name: f"{user_name}@alleninstitute.org",
) -> None:
    """
    Initializes the WatchdogDataTransferService.

    Args:
        source: The source directory or file to monitor
        settings: WatchdogSettings containing destination and configuration options
        validate: Whether to validate the project name
        session_name: Name of the session
        ui_helper: UI helper for user prompts

    Example:
        ```python
        # Basic initialization:
        settings = WatchdogSettings(
            destination="//server/archive/session_001",
            project_name="behavior_project"
        )
        service = WatchdogDataTransferService(
            source="C:/data/session_001",
            settings=settings
        )

        # Advanced configuration:
        settings = WatchdogSettings(
            destination="//server/archive/session_001",
            project_name="behavior_project",
            schedule_time=datetime.time(hour=23),
            platform=Platform.BEHAVIOR,
            force_cloud_sync=True,
            delete_modalities_source_after_success=True,
            extra_identifying_info={"experiment_type": "foraging"}
        )
        service = WatchdogDataTransferService(
            source="C:/data/session_001",
            settings=settings
        )
        ```
    """
    self._settings = settings
    self._source = source

    self._aind_session_data_mapper: Optional[TSessionMapper] = None

    _default_exe = os.environ.get("WATCHDOG_EXE", None)
    _default_config = os.environ.get("WATCHDOG_CONFIG", None)

    if _default_exe is None or _default_config is None:
        raise ValueError("WATCHDOG_EXE and WATCHDOG_CONFIG environment variables must be defined.")

    self.executable_path = Path(_default_exe)
    self.config_path = Path(_default_config)

    self._watch_config: Optional[WatchConfig] = None
    self._manifest_config: Optional[ManifestConfig] = None

    self._validate_project_name = validate

    if validate:
        self.validate()

    self._watch_config = WatchConfig.model_validate(self._read_yaml(self.config_path))

    self._ui_helper = ui_helper or ui.DefaultUIHelper()
    self._session_name = session_name
    self._email_from_experimenter_builder = email_from_experimenter_builder

aind_session_data_mapper property

aind_session_data_mapper: TSessionMapper

Gets the aind-data-schema session data mapper.

Returns:

Type	Description
TSessionMapper	The session data mapper

Raises:

Type	Description
ValueError	If the data mapper is not set

settings property

settings: TSettings

Returns the settings for the data transfer service.

with_aind_session_data_mapper

with_aind_session_data_mapper(
    value: TSessionMapper,
) -> WatchdogDataTransferService[TSessionMapper]

Sets the aind-data-schema session data mapper.

Parameters:

Name	Type	Description	Default
value	TSessionMapper	The data mapper to set	required

Raises:

Type	Description
ValueError	If the provided value is not a valid data mapper

Source code in src/clabe/data_transfer/aind_watchdog.py

def with_aind_session_data_mapper(self, value: TSessionMapper) -> "WatchdogDataTransferService[TSessionMapper]":
    """
    Sets the aind-data-schema session data mapper.

    Args:
        value: The data mapper to set

    Raises:
        ValueError: If the provided value is not a valid data mapper
    """
    self._aind_session_data_mapper = value
    return self

transfer

transfer() -> None

Executes the data transfer by generating a Watchdog manifest configuration.

Creates and deploys a manifest configuration file that the watchdog service will use to monitor and transfer data according to the specified parameters.

Source code in src/clabe/data_transfer/aind_watchdog.py

def transfer(self) -> None:
    """
    Executes the data transfer by generating a Watchdog manifest configuration.

    Creates and deploys a manifest configuration file that the watchdog service
    will use to monitor and transfer data according to the specified parameters.
    """
    try:
        if not self.is_running():
            logger.warning("Watchdog service is not running. Attempting to start it.")
            try:
                self.force_restart(kill_if_running=False)
            except subprocess.CalledProcessError as e:
                logger.error("Failed to start watchdog service. %s", e)
                raise RuntimeError("Failed to start watchdog service.") from e
            else:
                if not self.is_running():
                    logger.error("Failed to start watchdog service.")
                    raise RuntimeError("Failed to start watchdog service.")
                else:
                    logger.info("Watchdog service restarted successfully.")

        logger.info("Creating watchdog manifest config.")

        if not self.aind_session_data_mapper.is_mapped():
            raise ValueError("Data mapper has not been mapped yet.")

        self._manifest_config = self._create_manifest_config_from_ads_session(
            ads_session=self.aind_session_data_mapper.mapped,
            session_name=self._session_name,
        )

        if self._watch_config is None:
            raise ValueError("Watchdog config is not set.")

        assert self._manifest_config.name is not None, "Manifest config name must be set."
        _manifest_path = self.dump_manifest_config(
            path=Path(self._watch_config.flag_dir) / self._manifest_config.name
        )
        logger.info("Watchdog manifest config created successfully at %s.", _manifest_path)

    except (pydantic.ValidationError, ValueError, IOError) as e:
        logger.error("Failed to create watchdog manifest config. %s", e)
        raise e

validate

validate() -> bool

Validates the Watchdog service and its configuration.

Checks for required executables, configuration files, service status, and project name validity.

Returns:

Type	Description
bool	True if the service is valid, False otherwise

Raises:

Type	Description
FileNotFoundError	If required files are missing
HTTPError	If the project name validation fails

Source code in src/clabe/data_transfer/aind_watchdog.py

def validate(self) -> bool:
    """
    Validates the Watchdog service and its configuration.

    Checks for required executables, configuration files, service status,
    and project name validity.

    Returns:
        True if the service is valid, False otherwise

    Raises:
        FileNotFoundError: If required files are missing
        HTTPError: If the project name validation fails
    """
    logger.info("Attempting to validate Watchdog service.")
    if not self.executable_path.exists():
        raise FileNotFoundError(f"Executable not found at {self.executable_path}")
    if not self.config_path.exists():
        raise FileNotFoundError(f"Config file not found at {self.config_path}")

    if not self.is_running():
        logger.warning(
            "Watchdog service is not running. \
                            After the session is over, \
                            the launcher will attempt to forcefully restart it"
        )
        return False

    if self.settings.project_name is None:
        logger.warning("Watchdog project name is not set. Skipping validation.")
    else:
        try:
            _valid_proj = self.is_valid_project_name()
            if not _valid_proj:
                logger.warning("Watchdog project name is not valid.")
        except HTTPError as e:
            logger.error("Failed to fetch project names from endpoint. %s", e)
            raise e
        return _valid_proj

    return True

is_valid_project_name

is_valid_project_name() -> bool

Checks if the project name is valid by querying the metadata service.

Validates the project name against the list of known projects from the AIND metadata service.

Returns:

Type	Description
bool	True if the project name is valid, False otherwise

Source code in src/clabe/data_transfer/aind_watchdog.py

def is_valid_project_name(self) -> bool:
    """
    Checks if the project name is valid by querying the metadata service.

    Validates the project name against the list of known projects from
    the AIND metadata service.

    Returns:
        True if the project name is valid, False otherwise
    """
    project_names = self._get_project_names()
    return self._settings.project_name in project_names

is_running

is_running() -> bool

Checks if the Watchdog service is currently running.

Uses system process monitoring to determine if the watchdog executable is currently active.

Returns:

Type	Description
bool	True if the service is running, False otherwise

Example

# Check service status:
settings = WatchdogSettings(
    destination="//server/data",
    project_name="my_project"
)
service = WatchdogDataTransferService(source="C:/data", settings=settings)
if service.is_running():
    print("Watchdog service is active")
else:
    print("Watchdog service is not running")
    service.force_restart()

Source code in src/clabe/data_transfer/aind_watchdog.py

def is_running(self) -> bool:
    """
    Checks if the Watchdog service is currently running.

    Uses system process monitoring to determine if the watchdog executable
    is currently active.

    Returns:
        True if the service is running, False otherwise

    Example:
        ```python
        # Check service status:
        settings = WatchdogSettings(
            destination="//server/data",
            project_name="my_project"
        )
        service = WatchdogDataTransferService(source="C:/data", settings=settings)
        if service.is_running():
            print("Watchdog service is active")
        else:
            print("Watchdog service is not running")
            service.force_restart()
        ```
    """
    output = subprocess.check_output(
        ["tasklist", "/FI", f"IMAGENAME eq {self.executable_path.name}"], shell=True, encoding="utf-8"
    )
    processes = [line.split()[0] for line in output.splitlines()[2:]]
    return len(processes) > 0

force_restart

force_restart(kill_if_running: bool = True) -> Popen[bytes]

Attempts to restart the Watchdog application.

Terminates the existing service if running and starts a new instance with the current configuration.

Parameters:

Name	Type	Description	Default
kill_if_running	bool	Whether to terminate the service if it's already running	True

Returns:

Type	Description
Popen[bytes]	A subprocess.Popen object representing the restarted service

Source code in src/clabe/data_transfer/aind_watchdog.py

def force_restart(self, kill_if_running: bool = True) -> subprocess.Popen[bytes]:
    """
    Attempts to restart the Watchdog application.

    Terminates the existing service if running and starts a new instance
    with the current configuration.

    Args:
        kill_if_running: Whether to terminate the service if it's already running

    Returns:
        A subprocess.Popen object representing the restarted service
    """
    if kill_if_running is True:
        while self.is_running():
            subprocess.run(["taskkill", "/IM", self.executable_path.name, "/F"], shell=True, check=True)

    cmd_factory = "{exe} -c {config}".format(exe=self.executable_path, config=self.config_path)

    return subprocess.Popen(cmd_factory, start_new_session=True, shell=True)

dump_manifest_config

dump_manifest_config(
    path: Optional[PathLike] = None, make_dir: bool = True
) -> Path

Dumps the manifest configuration to a YAML file.

Saves the current manifest configuration to a file that can be processed by the watchdog service.

Parameters:

Name	Type	Description	Default
path	Optional[PathLike]	The file path to save the manifest	None
make_dir	bool	Whether to create the directory if it doesn't exist	True

Returns:

Type	Description
Path	The path to the saved manifest file

Raises:

Type	Description
ValueError	If the manifest or watch configuration is not set

Source code in src/clabe/data_transfer/aind_watchdog.py

def dump_manifest_config(self, path: Optional[os.PathLike] = None, make_dir: bool = True) -> Path:
    """
    Dumps the manifest configuration to a YAML file.

    Saves the current manifest configuration to a file that can be
    processed by the watchdog service.

    Args:
        path: The file path to save the manifest
        make_dir: Whether to create the directory if it doesn't exist

    Returns:
        The path to the saved manifest file

    Raises:
        ValueError: If the manifest or watch configuration is not set
    """
    manifest_config = self._manifest_config
    watch_config = self._watch_config

    if manifest_config is None or watch_config is None:
        raise ValueError("ManifestConfig or WatchConfig config is not set.")

    path = (Path(path) if path else Path(watch_config.flag_dir) / f"manifest_{manifest_config.name}.yaml").resolve()

    if path.suffix not in [".yml", ".yaml"]:
        path = path.with_suffix(".yaml")

    if not path.name.startswith("manifest_"):
        logger.info("Prefix manifest_ not found in file name. Appending it.")
        path = path.with_name(f"manifest_{path.stem}{path.suffix}")

    if make_dir and not path.parent.exists():
        path.parent.mkdir(parents=True, exist_ok=True)

    manifest_config.destination = Path(manifest_config.destination)
    manifest_config.schemas = [Path(schema) for schema in manifest_config.schemas]
    for modality in manifest_config.modalities:
        manifest_config.modalities[modality] = [_path for _path in manifest_config.modalities[modality]]

    self._write_yaml(manifest_config, path)
    return path

prompt_input

prompt_input() -> bool

Prompts the user to confirm whether to generate a manifest.

Provides user interaction to confirm manifest generation for the watchdog service.

Returns:

Type	Description
bool	True if the user confirms, False otherwise

Example

# Interactive manifest generation:
settings = WatchdogSettings(
    destination="//server/data",
    project_name="my_project"
)
service = WatchdogDataTransferService(source="C:/data", settings=settings)
if service.prompt_input():
    service.transfer()
    print("Manifest generation confirmed")
else:
    print("Manifest generation cancelled")

Source code in src/clabe/data_transfer/aind_watchdog.py

def prompt_input(self) -> bool:
    """
    Prompts the user to confirm whether to generate a manifest.

    Provides user interaction to confirm manifest generation for the
    watchdog service.

    Returns:
        True if the user confirms, False otherwise

    Example:
        ```python
        # Interactive manifest generation:
        settings = WatchdogSettings(
            destination="//server/data",
            project_name="my_project"
        )
        service = WatchdogDataTransferService(source="C:/data", settings=settings)
        if service.prompt_input():
            service.transfer()
            print("Manifest generation confirmed")
        else:
            print("Manifest generation cancelled")
        ```
    """
    return self._ui_helper.prompt_yes_no_question("Would you like to generate a watchdog manifest (Y/N)?")

build_runner classmethod

build_runner(
    settings: WatchdogSettings,
    aind_session_data_mapper: Promise[
        [Launcher], TSessionMapper
    ]
    | TSessionMapper,
    **kwargs,
) -> Callable[
    [Launcher], WatchdogDataTransferService[TSessionMapper]
]

A factory method for creating the watchdog service.

Parameters:

Name	Type	Description	Default
settings	WatchdogSettings	The watchdog settings.	required
aind_session_data_mapper	Promise[[Launcher], TSessionMapper] | TSessionMapper	The aind session data mapper.	required

Returns:

Type	Description
Callable[[Launcher], WatchdogDataTransferService[TSessionMapper]]	A factory for WatchdogDataTransferService.

Source code in src/clabe/data_transfer/aind_watchdog.py

@classmethod
def build_runner(
    cls,
    settings: WatchdogSettings,
    aind_session_data_mapper: Promise[[Launcher], TSessionMapper] | TSessionMapper,
    **kwargs,
) -> Callable[[Launcher], "WatchdogDataTransferService[TSessionMapper]"]:
    """
    A factory method for creating the watchdog service.

    Args:
        settings: The watchdog settings.
        aind_session_data_mapper: The aind session data mapper.

    Returns:
        A factory for WatchdogDataTransferService.
    """

    def _from_launcher(
        launcher: Launcher,
    ) -> "WatchdogDataTransferService":
        """Inner callable to create the service from a launcher"""
        _aind_session_data_mapper = (
            aind_session_data_mapper.result
            if isinstance(aind_session_data_mapper, Promise)
            else aind_session_data_mapper
        )

        if not _aind_session_data_mapper.is_mapped():
            raise ValueError("Data mapper has not mapped yet. Cannot create watchdog.")

        _settings = settings.model_copy()

        _session = launcher.get_session(strict=True)
        _settings.destination = Path(_settings.destination) / _session.subject
        launcher.copy_logs()
        service = cls(
            source=launcher.session_directory, settings=_settings, session_name=_session.session_name, **kwargs
        ).with_aind_session_data_mapper(_aind_session_data_mapper)
        service.transfer()
        return service

    return _from_launcher