Inhooks

Lightweight Incoming Webhooks Gateway

5 min readMay 19, 2023

This blog has moved to https://didil.substack.com/

Introduction

When subscribing to Webhooks from third parties, developers usually need their services to handle failures, retries, delayed processing, buffering, etc. Inhooks aims to be an open source, lightweight, incoming webhooks gateway solution to handle these requirements in a simple but scalable way. Written in Go and runnable as a single binary or docker container. It only requires a Redis database for storage/queueing.

Architecture

High level overview

Inhooks listens to HTTP webhooks and saves the messages to Redis. A processing module retrieves the messages and sends them reliably to the defined targets.

Features

Receive HTTP Webhooks and save them to a Redis database
Fanout messages to multiple HTTP targets
Fast, concurrent processing
Supports delayed processing
Supports retries on failure with configurable number of attempts, interval and constant or exponential backoff
… more features planned

Demo

Let’s now see inhooks in action. For this simple demo, we will run inhooks on a cloud virtual machine and we will send it Github push event Webhooks.

We have previously installed Redis on the virtual machine. For this demo we’re running Redis on the same server but connecting to any remote Redis instance should work.

Inhooks Setup

Let’s first fetch and extract the inhooks binary tarball:

$ wget -qO - https://github.com/didil/inhooks/releases/download/v0.1.5/inhooks_0.1.5_linux_amd64.tar.gz | tar -xzvf -
CHANGELOG.md
LICENSE
README.md
inhooks

Next we create the inhooks config file inhooks.yml in the same directory:

#inhooks.yml
flows:
  - id: github-webhook
    source:
      id: gh
      slug: github
      type: http
    sinks:
      - id: requestbin-simple
        type: http
        url: https://en87n9uqqlch.x.pipedream.net
      - id: requestbin-advanced
        type: http
        url: https://ene9c6929em87.x.pipedream.net
        delay: 60s # delay processing by 60 seconds
        retryInterval: 5m # on error, retry after 5 minutes
        retryExpMultiplier: 2 # exponential backoff with a multiplier of 2
        maxAttempts: 10 # try up to 10 times

With this configuration, inhooks will ingest http POST requests sent to the path /api/v1/ingest/github and

Send a request to the first requestbin endpoint immediately.
Send a request to the second requestbin endpoint after a 60s delay. The second sink (which is what we call the output targets in inhooks) is also configured to retry with exponential backoff starting at 5 minutes with 10 maximum attempts.

Github Webhooks Setup

We add the ingest url endpoint http://35.232.6.40:4567/api/v1/ingest/github to our github test repository’s webhooks configuration and we select the content type as application/json.

Running the server

We run the server and set a few environment variables. Here we set APP_ENV to development to get human-readable log messages, but in a production environment we would usually set it to production.

$ APP_ENV=development PORT=4567 REDIS_URL=redis://localhost:6379 REDIS_INHOOKS_DB_NAME=demo ./inhooks 
2023-05-19T14:30:07.378Z INFO api/main.go:45 loading inhooks config {"inhooksConfigFile": "inhooks.yml"}
2023-05-19T14:30:07.378Z INFO services/inhooks_config_service.go:109 loaded flow {"id": "github-webhook", "sourceID": "gh", "sourceSlug": "github", "sourceType": "http"}
2023-05-19T14:30:07.378Z INFO services/inhooks_config_service.go:117 flow sink {"id": "requestbin-simple", "type": "http", "url": "https://en87n9uqqlch.x.pipedream.net", "delay": "0s"}
2023-05-19T14:30:07.378Z INFO services/inhooks_config_service.go:117 flow sink {"id": "requestbin-advanced", "type": "http", "url": "https://ene9c6929em87.x.pipedream.net", "delay": "1m0s"}
2023-05-19T14:30:07.380Z INFO api/main.go:87 listening ... {"addr": ":4567"}
2023-05-19T14:30:07.380Z INFO api/main.go:124 starting supervisor ...

The server has started successfully. We can now trigger a git push on our test Github repository and see the results in the logs:

2023-05-19T14:31:53.689Z INFO handlers/ingest.go:20 new ingest request {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github"}
2023-05-19T14:31:53.691Z INFO handlers/ingest.go:53 message queued {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github", "flowID": "github-webhook", "sourceID": "gh", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "queue": "ready"}
2023-05-19T14:31:53.691Z INFO handlers/ingest.go:53 message queued {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github", "flowID": "github-webhook", "sourceID": "gh", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "queue": "scheduled", "nextAttemptAfter": "2023-05-19T14:32:53.690Z"}
2023-05-19T14:31:53.691Z INFO supervisor/ready.go:65 processing message {"flowID": "github-webhook", "sinkID": "requestbin-simple", "sinkType": "http", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001", "attempt#": 1}
2023-05-19T14:31:53.691Z INFO handlers/ingest.go:57 ingest request succeeded {"reqID": "inhooks-demo-1/YGyDi2qlNy-000001", "sourceSlug": "github", "flowID": "github-webhook", "sourceID": "gh"}
2023-05-19T14:31:53.870Z INFO supervisor/ready.go:77 message processed ok {"flowID": "github-webhook", "sinkID": "requestbin-simple", "sinkType": "http", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}
2023-05-19T14:31:53.871Z INFO supervisor/ready.go:83 message processed ok - finalized {"flowID": "github-webhook", "sinkID": "requestbin-simple", "sinkType": "http", "messageID": "382f14fc-5317-480e-9ec0-4f153e69dd9b", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}

What happened above is :

An ingest request was received.
The message was queued to redis for both the requestbin-simple as well as the requestbin-advanced sinks.
For the requestbin-simple sink, the message was sent to requestbin and the request succeeded.
For the requestbin-advanced sink, the message was scheduled to be processed after 60s as specified in the config.

After a couple of minutes we can see in the logs:

2023-05-19T14:33:07.389Z INFO supervisor/ready.go:65 processing message {"flowID": "github-webhook", "sinkID": "requestbin-advanced", "sinkType": "http", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001", "attempt#": 1}
2023-05-19T14:33:07.550Z INFO supervisor/ready.go:77 message processed ok {"flowID": "github-webhook", "sinkID": "requestbin-advanced", "sinkType": "http", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}
2023-05-19T14:33:07.550Z INFO supervisor/ready.go:83 message processed ok - finalized {"flowID": "github-webhook", "sinkID": "requestbin-advanced", "sinkType": "http", "messageID": "8921fc47-207a-4a33-a808-a06f2efd38f5", "ingestedReqID": "inhooks-demo-1/YGyDi2qlNy-000001"}

The message for the requestbin-advanced sink was processed and the request succeeded.

Let’s now have a look at the results on the RequestBin side.

For our first target:

And the second target:

The http headers and body look good. Seems like it worked 🎆 !

Conclusion

I hope you will find inhooks useful. This project is still very young but I believe it has a lot of potential and there are many useful features that could be added. If your backend services receive webhooks, please give inhooks a try in your dev/staging environment and share any feedback to https://github.com/didil/inhooks/issues.