Following up on my last post about a base self-configuring service class, horizontally scalable services need remote management functions.
- Authentication – Determine if a management client is permitted to connect and issue commands
- Remote Creation – A service daemon lists for creation requests and creates service instances
- Remote Shutdown – The service itself should be able to shut itself down in response to a management command (either because the cluster is shutting down, the software needs to be upgraded, or the service load is low enough to not be needed)
- Load Reporting – When queried, a service should be able to report its load to a client (such as a dashboard monitor)
- Alerts – When certain load thresholds or error rates are exceeded, a service needs to be able to report an alert condition
- Metrics – in addition to load reporting and raising alerts to a monitor, each service should be responsible for providing detailed metrics (generally, these should be logged to a RDB rather than a log file)
This implies dashboard (web) and management (C/S or web) clients. Service management is a backchannel between all services, including those that may not normally respond to network clients (to use a bad example, a log archiver).
Some services are implied by these requirements:
- Management Service – Collects metrics and alerts, provides the dashboard
- Watchdog Service – Makes sure critical services are healthy and restarts them on failure, listens for service creation requests
Finally, a main channel for communication between services should be available, although not all services will take advantage of it. A Message Queue service receives service-to-service messages in a cluster and ensures consistency and delivery. I’ll cover the design for clustered services and the message queue in another post.