Getting Started

Welcome to the Magistrala Developer Guide! This guide provides comprehensive instructions for installing, building, running, and interacting with Magistrala, an advanced messaging platform. Follow the steps below to get started with Magistrala development, understand its architecture, and interact with the system.

Fast Start

To spin up Magistrala, run the following commands:

git clone https://github.com/absmach/magistrala.git
cd magistrala
export MG_UI_DOCKER_ACCEPT_EULA=yes
make run_stable args=-d

Once the services are running, open http://localhost:3000 in your browser to view the UI. This is the Magistrala web-based platform, where you can manage clients, channels, messages, and view system activity in real time.

This runs Magistrala with the UI using prebuilt Docker images.

You can now continue below to:

• Provision the system
• Send messages
• Use the CLI

If you're interested in building from source, see Developer Setup section below.

Developer Setup

Step 1 - Install Magistrala

Clone the Repository

Magistrala source can be found in the official Magistrala GitHub repository. You should fork this repository in order to make changes to the project. The forked version of the repository should be cloned using the following:

git clone <forked repository> $SOMEPATH/magistrala
cd $SOMEPATH/magistrala

Note: If your $SOMEPATH is equal to $GOPATH/src/github.com/absmach/magistrala, make sure that your $GOROOT and $GOPATH do not overlap (otherwise, go modules won't work).

Step 2 - Prerequisites

Before building Magistrala, install the following tools:

• Docker (version 20.10.16 or later)
• Docker compose (version 1.29.2 or later)
• Go (version 1.19.4 or higher)
• Go Protobuf whose installation instructions are here
• C++ protobuf since Go Protobuf uses C bindings
• GNU Make (for build automation)

Environment Setup:
Ensure that $GOPATH/bin is added to your system’s $PATH.

Step 3 - Build Magistrala

Build All Services

Use the GNU Make tool to build all Magistrala services:

make

The response will be:

CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:51:59' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/bootstrap cmd/bootstrap/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:10' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/provision cmd/provision/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:11' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/re cmd/re/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:14' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/postgres-writer cmd/postgres-writer/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:15' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/postgres-reader cmd/postgres-reader/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:16' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/timescale-writer cmd/timescale-writer/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:17' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/timescale-reader cmd/timescale-reader/main.go
CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:52:18' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/cli cmd/cli/main.go

Build artifacts will be put in the build directory.

N.B. All Magistrala services are built as a statically linked binaries. This way they can be portable (transferred to any platform just by placing them there and running them) as they contain all needed libraries and do not relay on shared system libraries. This helps creating FROM scratch dockers.

Build Individual Microservice

Individual microservices can be built with:

make <microservice_name>

<microservice_name> must be one of the names in the Makefile's SERVICES list — currently atom-bootstrap, certs, postgres-writer, postgres-reader, timescale-writer, timescale-reader, and fluxmq. Services outside this list (e.g. auth, which still has a cmd/auth entry point) can be built directly with go build -o build/<name> cmd/<name>/main.go.

For example:

make atom-bootstrap

will build the Atom Bootstrap microservice.

The response will be:

CGO_ENABLED=0 GOOS= GOARCH=amd64 GOARM= go build -tags nats --tags nats -ldflags "-s -w -X 'github.com/absmach/magistrala.BuildTime=2025-02-11_14:54:15' -X 'github.com/absmach/magistrala.Version=unknown' -X 'github.com/absmach/magistrala.Commit=ddc43c482f6c98f3a4d49aa1d609bfae9e0e7d34'" -o build/atom-bootstrap cmd/atom-bootstrap/main.go

Build Dockers

Dockers can be built with:

make dockers

or individually with:

make docker_<microservice_name>

For example:

make docker_atom-bootstrap

N.B. Magistrala creates FROM scratch docker containers which are compact and small in size.

N.B. The clients-db and users-db containers are built from a vanilla PostgreSQL docker image downloaded from docker hub which does not persist the data when these containers are rebuilt. Thus, rebuilding of all docker containers with make dockers or rebuilding the clients-db and users-db containers separately with make docker_clients-db and make docker_users-db respectively, will cause data loss. All your users, clients, channels and connections between them will be lost! As we use this setup only for development, we don't guarantee any permanent data persistence. Though, in order to enable data retention, we have configured persistent volumes for each container that stores some data. If you want to update your Magistrala dockerized installation and want to keep your data, use make cleandocker to clean the containers and images and keep the data (stored in docker persistent volumes) and then make run to update the images and the containers. Check the Cleaning up your dockerized Magistrala setup section for details. Please note that this kind of updating might not work if there are database changes.

Building Docker images for development

In order to speed up build process, you can use commands such as:

make dockers_dev

or individually with

make docker_dev_<microservice_name>

Commands make dockers and make dockers_dev are similar. The main difference is that building images in the development mode is done on the local machine, rather than an intermediate image, which makes building images much faster. Before running this command, corresponding binary needs to be built in order to make changes visible. This can be done using make or make <service_name> command. Commands make dockers_dev and make docker_dev_<service_name> should be used only for development to speed up the process of image building. For deployment images, commands from section above should be used.

Step 4 – Run Magistrala

To run Magistrala with its UI components, you must accept the Absmach End User License Agreement (EULA).

This EULA applies only to the Magistrala UI services.

The Magistrala core remains free and open-source, licensed under the Apache 2.0 License.

This step is optional and only required if you want to use the UI services.

To accept the UI EULA, set the following environment variable:

export MG_UI_DOCKER_ACCEPT_EULA=yes

Once everything is installed and built, execute the following command from the project root:

make run_latest

Or, to prevent Docker logs from flooding the terminal:

make run_latest args=-d

Quick summary

Assuming all required tools are installed and the necessary ports are available, you can run Magistrala with just four commands:

git clone https://github.com/absmach/magistrala.git
cd magistrala
export MG_UI_DOCKER_ACCEPT_EULA=yes
make run_latest args=-d

Go to http://localhost:3000 to start using UI.

(Corrected from a bare make run, which is not a target in the current Makefile — verified run_latest, run_latest_ci, run_stable, run_live, run_tls, run_addons, and rundev all exist; there is no plain run.)

Suggested workflow

When the project is first cloned to your system, you will need to make sure and build all of the Magistrala services.

make
make dockers_dev

As you develop and test changes, only the services related to your changes will need to be rebuilt. This will reduce compile time and create a much more enjoyable development experience.

make <microservice_name>
make docker_dev_<microservice_name>
make run_latest

Overriding the default docker-compose configuration

Sometimes, depending on the use case and the user's needs it might be useful to override or add some extra parameters to the docker-compose configuration. These configuration changes can be done by specifying multiple compose files with the docker-compose command line option -f as described here. The following format of the docker-compose command can be used to extend or override the configuration:

docker-compose -f docker/docker-compose.yml -f docker/docker-compose.custom1.yml -f docker/docker-compose.custom2.yml up [-d]

In the command above each successive file overrides the previous parameters.

A practical example in our case would be overriding one of the FluxMQ broker nodes — say, to disable its restart policy while you're actively debugging it.

docker-compose.fluxmq-debug.yml

services:
  fluxmq-node1:
    restart: "no"

When we have the override files in place, to compose the whole infrastructure including the persistent volumes we can execute:

docker-compose -f docker/docker-compose.yml -f docker/docker-compose.fluxmq-debug.yml up -d

Note: Please store your customizations to some folder outside the Magistrala's source folder and maybe add them to some other git repository. You can always apply your customizations by pointing to the right file using docker-compose -f ....

Cleaning up your dockerized Magistrala setup

If you want to clean your whole dockerized Magistrala installation you can use the make pv=true cleandocker command. Please note that by default the make cleandocker command will stop and delete all of the containers and images, but NOT DELETE persistent volumes. If you want to delete the gathered data in the system (the persistent volumes) please use the following command make pv=true cleandocker (pv = persistent volumes). This form of the command will stop and delete the containers, the images and will also delete the persistent volumes.

Step 5 - Install the CLI

The Magistrala CLI is the primary interface for interacting with the system.

Open a new terminal from which you can interact with the running Magistrala system. The Magistrala releases page is available at latest CLI release, but recent releases do not publish CLI tarball assets. Build the CLI from source as described in the CLI guide.

Make sure that $GOBIN is added to your $PATH so that magistrala-cli command can be accessible system-wide

Build magistrala-cli

Build magistrala-cli by following the CLI guide.

Step 6 - Provision the System

Provision a workspace with devices and channels using the CLI:

magistrala-cli workspaces create "Acme IoT" --token $TOKEN
magistrala-cli devices create '{"name":"Temperature Sensor 1"}' <workspace_id> --token $TOKEN
magistrala-cli channels create <workspace_id> "Telemetry Channel" --token $TOKEN

See the Workspaces, Devices, and Channels CLI guides for the full set of options.

Step 7 - Interact with Magistrala

With Magistrala running, you can now send messages.

Send Messages

Once a device is connected to a channel, it can publish messages over any of the platform's protocol adapters (HTTP, MQTT, CoAP, WebSocket). For example, over HTTP:

curl -sSiX POST http://localhost/http/m/<workspace_id>/c/<channel_id> \
-H "Content-Type: application/senml+json" \
-H "Authorization: Client <device_secret>" \
-d '[{"bn":"some-base-name:","bt":1.276020076001e+09, "bu":"A","bver":5, "n":"voltage","u":"V","v":120.1}, {"n":"current","t":-5,"v":1.2}, {"n":"current","t":-4,"v":1.3}]'

See Messaging for the other protocol adapters, and the API reference for the full Messages API.

Managing Microservices

Running of the Magistrala microservices can be tricky, as there is a lot of them and each demand configuration in the form of environment variables.

The whole system (set of microservices) can be run with one command:

make rundev

which will properly configure and run all microservices.

Please assure that MQTT microservice has node_modules installed.

N.B. make rundev actually calls helper script scripts/run.sh, so you can inspect this script for the details.

Testing

To run all of the tests you can execute:

make test

Dockertest is used for the tests, so to run them, you will need the Docker daemon/service running.

Installing

Installing Go binaries is simple: just move them from build to $GOBIN (do not fortget to add $GOBIN to your $PATH).

You can execute:

make install

which will do this copying of the binaries.

N.B. Only Go binaries will be installed this way. The MQTT broker is a written in Erlang and its build scripts can be found in docker/vernemq dir.

Cross-compiling for ARM

Magistrala can be compiled for ARM platform and run on Raspberry Pi or other similar IoT gateways, by following the instructions here or here as well as information found here. The environment variables GOARCH=arm and GOARM=7 must be set for the compilation.

Cross-compilation for ARM with Magistrala make:

GOOS=linux GOARCH=arm GOARM=7 make

Message Broker

FluxMQ is Magistrala's message broker and event stream, handling MQTT, WebSocket, HTTP, CoAP, and AMQP protocol adapters — see Architecture for the full model.

MQTT Broker

To build Magistrala MQTT message broker Docker image, use the following commands:

cd docker/vernemq
docker build --no-cache . -t magistrala/vernemq

The Magistrala uses the VerneMQ for implementation of the MQTT messaging. Therefore, for some questions or problems you can also check out the VerneMQ documentation or reach out its contributors.

Database Configuration

PostgreSQL

Magistrala uses PostgreSQL to store metadata (users, clients and channels entities alongside with authorization tokens). It expects that PostgreSQL DB is installed, set up and running on the local system.

Information how to set-up (prepare) PostgreSQL database can be found here, and it is done by executing following commands:

# Create `users` and `clients` databases
sudo -u postgres createdb users
sudo -u postgres createdb clients

# Set-up Postgres roles
sudo su - postgres
psql -U postgres
postgres=# CREATE ROLE magistrala WITH LOGIN ENCRYPTED PASSWORD 'magistrala';
postgres=# ALTER USER magistrala WITH LOGIN ENCRYPTED PASSWORD 'magistrala';

Deployment

Deployment Prerequisites

Magistrala depends on several infrastructural services, notably the default message broker, FluxMQ, and the PostgreSQL database.

Troubleshooting

Common Issues

1. Docker Containers Not Starting

   • Ensure Docker and Docker Compose are installed and running.

2. CLI Commands Not Recognized

   • Confirm that $GOBIN is in your system's $PATH.

3. Database Connection Issues

   • Verify PostgreSQL is running and the magistrala role exists.

Conclusion

With Magistrala installed and running, you can now explore its messaging capabilities, test interactions, and extend the system as needed. For authorization and access control, see Authorization.