1. About WebPlug

Webplug is a webhooks as a service infrastructure that aims to provide a template for managing webhooks for users of a service.

You would typically create a service which can be regarded as an application and then you can register event types that your users may be interested in receiving notifications for. Let's say you are a payment provider and you provide services for businesses to create cards for their users. You would have some event types that a user can subscribe to; for example, card_issued, card_debited or card_credited. Whenever a user performs an operation on their card, you want the business to receive a notification at whatever endpoint they have subscribed to for each event type, so that they can then perform extra business logic for each user. This is a typical use case for webplug.

Webplug is built on top of AWS and provides a CLI app for you to deploy the infrastructure to an AWS account of your choice, there's also a command to destroy the infrastructure off your AWS account. Lastly we provide a management UI built with shadcn/tailwind you can modify based on your requirements

2. What are webhooks?

Webhooks are a mechanism for automating the flow of information between web applications in real-time. They allow one application to send data to another in a push-based manner, triggering events and enabling real-time communication. Webhooks are commonly used in scenarios where immediate or near real-time updates are required, such as in notifications, integrations, and automation workflows.

2.1. Alternatives to webhooks

Webhoooks are not the only approach to real-time communication in software applications. There are alternatice technologies and patterns that serve similar purposes. One of the major differences between webhooks and other types of real real-time communication types is that webhooks are omni-directional. This section outlines a few of them:

• Web-Sockets: Web Sockets provide full-duplex communication channels over a single, long-lived connection. They are designed for real-time bidirectional communication, making them suitable for applications requiring low-latency updates. Some of the use cases include: real-time chat applications, live updated, collaborative editing.
• Polling: Polling involves regularly checking a server for updates. Clients make periodic requeses to the server, asking if there are any new events or data. Long polling is a variant where the server holds the request open until new data is available.
• Server Side Events: SSE is a simple and efficient mechanism for sending updates from a server to a client over a single HTTP connection. It allows servers to push data to web clients over HTTP in real-time. Some of the use cases include: News feeds, live score updates, stock tickers.

2.2 Examples of webhook services

There are a couple services that currently use webhooks on a large scale. A couple of them include:

• Github Webhooks: Github provides webhooks to notify external systems about events in a repository. For example, you can set up a webhooks to receive notifications about new commits, pull requests, or issues.

  This was the major inspiration behind webplug. I made use of github webhooks when setting up a CI/CD pipeline with AWS Codebuild and AWS Codepipeline and started to brainstorm on how I could abstract the functionality into a stand alone service.

• Stripe Webhooks: Stripe, a payment processing platform, uses webhooks to notify merchants about events related to payments, subscriptions, and customer actions. This allows businesses to take immediate actions based on payment events.
• Twilio Webhooks: Twilio, a cloud communications platform, offers webhooks to handle events related to phone calls, SMS messages, and other communication activities.

3. Webplug Architecture

3.1 Architecture diagram

3.2 Webplug stack

This section outlines all the technologies employed to bring webplug to life.

• AWS API Gateway: Provides a unified API for the client to interact with the backend service. Handles requests through the lambda functions.
• AWS Simple Notification Service: This is a fully managed messaging service from AWS. SNS was used to subscribe a new event type as a topic and then whenever a message is published, it would get sent to all subscribers of that topic.
• AWS Lambda functions: These were used to handle the several api routes used in the application.
• AWS Lambda layers: Lambda layer was used to initiate the pg pool connection and all lambda functions reference the connection.
• Cloudwatch Logs: SNS was configured to send the result of the message delivery to the cloudwatch logs of the log lambda. Whenever a new message result is received, the status is updated in the database.
• AWS CDK: AWS CDK was used to deploy the webplug infrastructure to AWS.
• Shadcn/tailwind: Shadcn/tailwind were used to develop components and style the management UI.

4. How does Webplug work?

This session explains everything that happens from client to server side communication.

Webplug mainly makes use of SNS to manage message delivery. The reason for this choice is because SNS can fanout messages to multiple subscribers and also automatically handles message delivery retries.

The steps involved in the process flow include:

• A new service is created
• A user is created and registered under the service
• Event types are created for the service. Whenever there is a request to create a new event type, we create an SNS topic to represent the event type.
• A user subscribes specific endpoints to particular event types. We retreive the SNS topic for the event type and create a new subscription with the endpoint.
• SNS sends a confirm subscription payload to the endpoint, you would need to copy the subscribe url and paste in a new tab to confirm the subscription so that future messages would be delivered.
• Whenever a new message is published for that event type, SNS would automatically send the message to all subscribers (endpoints) of that topic (event type).

5. Webplug JS Client

A JS client was built to expose methods to communicate with the various modules. This includes methods for the service, users, subscriptions, event types and events module.

The webplug JS client is installable from npm as shown in the documentation

6. Webplug CLI app

A cli app was built to deploy and destroy the infrastructure on your AWS account as well as spin up the management UI. Information on how to install and use the cli app is specified in the documentation.

The steps involved in deploying the infrastructure is as follows:

• The cdk code is cloned from the github repository
• The code for the management UI is cloned from the github repository
• The code for the lambda functions is cloned from the github repository
• When the deploy command is run, cdk references the lambda functions from the server directory which is where the cloned lambda functions are saved. Once the deployment is completed, cdk writes the api gateway url and api key id which can be used to retreive the api key itself to the outputs.json file. These variables are then saved to the .env file in the management UI codebase.
• The code for the management UI is cloned from the github repository

7. Webplug management UI

The management dashboard provides a UI to manage all your services, users, event types, subscriptions and view statuses of all messages sent to different subscribers (endpoints). Check out the documentation for how to access the management UI.

_{View information about the services in the webplug infrastructure.}

_{Create a new user and view users of a service.}

_{Create a new event type or view event types for a service.}

_{View subscriptions for a user.}

_{View messages sent to subscribers for all users.}

8. Future Considerations

These are some of the future work that can be implemented

• Give users the ability to rate limit their webhook endpoints.
• Implement functionality to manually trigger message retries after SNS fails to deliver the message after the maximum number of retries.
• Ensure orderly delivery of messages depending on the time they are created.
• Dead letter queue to hold messages that fail to be processed.