README
imonitor is a process supervisor or a process control system similar to the Supervisor. Just like its predecessor, it is not a substitute for init as "process id 1". Instead, it is used to control processes related to a project or a customer, and should be started like any other program at boot time. It is meant to be running inside of an OpenWrt.
Project Components
Project consists of following components:
- Application itself written in C programming language
- Docker Image in a form of a Dockerfile
- Integration tests written using CMocka
- Documentation in a Markdown format
Build Instructions
imonitor is written using C programming language and depends on a number of components found in OpenWrt for building and running.
To ease the building, testing, running and debugging of imonitor, Dockerfile, based on Ubuntu image, is available. This Dockerfile contains EVERYTHING you need for imonitor development.
But first, you need to build an Docker image. To build this image out of the Dockerfile, position yourself to the root of the source tree and execute bellow command:
docker build -t inteno/imonitor .
When built, use the following command to run a container:
docker run -d --name imonitor --privileged --rm -v ${PWD}:/opt/work -p 2222:22 -e LOCAL_USER_ID=`id -u $USER` inteno/imonitor:latest
The ${PWD} in above command will be automatically replaced by your current working directory. Again, ensure that you are positioned in the root of the source tree as you will need the contents of the whole project for building, testing and running of imonitor.
At this point, Docker container should be running in the background. To gain an access to the shell, execute following command:
docker exec --user=user -it imonitor bash
This will open up a shell and you will be positioned in the /opt/work directory, similar as bellow:
root@caeefca015e4:/opt/work#
Ensure that this directory contains all the project files.
Finally, build the imonitor inside of this running container.
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make
This will generate an executable with the filename of imonitor in the same directory that you are currently located, build. And that's it! Application is ready for usage.
Besides building, you can run and debug imonitor using the same container. After you are finished, you need to stop the container:
docker stop imonitor
Usage
imonitor requires a configuration file to function properly. The configuration file is an uci file. Sample configuration file (test/data/uci_file_valid) is provided.
Following lists all possible command-line flags:
$ ./imonitor -h
Usage: ./imonitor: <options>
Options:
-s <socket> path to ubus socket [/var/run/ubus/ubus.sock]
-c <uci_config_path> uci config file containing monitoring configuration [/etc/config/imonitor]
-h help
Example:
To run the application, execute following command:
./imonitor
If necessary, specify the path to the configuration file using the -c and path to the ubus socket using the -s flag.
Configuration File
Sample imonitor configuration files, test/data/uci_file_valid and test/data/uci_file_valid_execute_script, are provided.
Structure and Format
For convenience, here is the example configuration file:
config monitor
option app 'bsd'
option test 'bsd -s'
option stream 'stdout'
option string_match 'hangup'
option execute 'killall -9 bsd; sleep 1; bsd'
option interval '10'
option nr_tests '2'
config monitor
option app 'cat'
option test 'pidof cat | wc -w'
option stream 'stdout'
option string_match '0'
option execute './script.sh'
option interval '5'
option nr_tests '2'
As can be seen from the above example, the structure consists of one or more config groups called monitor. A monitor represents one logical monitoring item. For example, it can be used to tell the application to monitor a system daemon. Each monitor consists of options such as app and test, amongst others. An option holds a configuration parameter for the application. Each option has a name and a value.
Now onto available options. The following table lists all available options and their meaning.
| Option | Type | Description | Required | Default | Possible Values |
|---|---|---|---|---|---|
app |
string | Monitor or application name. It has no meaning to the application. | Yes | - | - |
test |
string | Shell command used for testing the condition. | Yes | - | - |
stream |
string | Standard stream from test command which is used to search for a pattern. |
Yes | - | stdout, stderr |
string_match |
string | stream is searched for exact match to this pattern. |
Yes* | - | - |
string_not_match |
string | Oposite of string_match, triggers only if stream is not exact match. |
Yes* | - | - |
regex_match |
string | stream is searched for a regex** match to this pattern. |
Yes* | - | - |
execute |
string | Shell command executed only when the test condition passes. | Yes | - | - |
interval |
integer | Periodic timer in seconds. | No | 5 | - |
nr_tests |
integer | Number of tests per interval if test condition is met. | No | 1 | - |
* - exactly one of these options is required ** - used regex flavor is POSIX ERE
Concepts and Workflow
imonitor is composed of two modules; parser and monitor. Parser is responsible for parsing of the uci configuration file. Refer to the Structure and Format. Monitor, as the name suggests, is responsible for health monitoring of a number of processes. If a monitor finds that some process is not running correctly as it is supposed to be, it will execute a custom shell command configured by a user. This could be a command to restart such zombi process, and/or even to send a report via mail or similar. This is entirely under a user control.
During startup, imonitor will parse the configuration file. In case of any error, imonitor will report it to the console and exit. Otherwise, a new timer will be started for each monitor based on interval. Then, testing will be conducted when a timer expires. Testing phase consists of the following:
- Shell command, configured with
testoption, is executed. Upon completion, this shell command will return thestdoutand/orstderr. - Contents of a configured
streamwill be checked if it matches the pattern specified instring_match,string_not_matchorregex_match. - In case of a match, testing phase will repeat itself for
nr_teststimes. - If a condition is still met after the
nr_tests, a shell command, configured with theexecuteoption, is finally executed.
It is worth to note that the test and execute options can be any shell command. In case when you have a complicated command set that you want to execute, you can write your own shell script as usual and put its absolute path in test and/or execute options. Just note that a script needs to write something to a stdout or stderr if you are planning to put it into test option to be useful. This is so that imonitor can then compare match pattern with the stream of your choice. As mentioned, you can also use a script on an execute option. But here, you don't need to write anything to stdout and/or stderr since the imonitor will ignore output from the execute command. See Structure and Format for an example.
This whole testing phase repeats for each interval, indefinitely.
Dependencies
Build-Time Dependencies
To successfully build imonitor, following libraries are needed:
| Dependency | Link | License |
|---|---|---|
| libuci | https://git.openwrt.org/project/uci.git | GPLv2 |
| libubox | https://git.openwrt.org/project/libubox.git | BSD |
| CMocka | https://cmocka.org/ | Apache License |
Note that CMocka is only needed when a debug build of imonitor is generated.
Run-Time Dependencies
In order to run the imonitor, following dependencies are needed to be running before imonitor.
| Dependency | Link | License |
|---|---|---|
| ubusd | https://git.openwrt.org/project/ubus.git | LGPL 2.1 |
| rpcd | https://git.openwrt.org/project/rpcd.git | ISC |
System daemons, ubusd and rpcd, are used to execute shell commands over ubus.
Static Source Code Analysis
imonitor comes with static source code analysis. To get the analysis report, execute following:
make check
If there are no issues, output from the analysis will be empty. Otherwise, above command will fail with the listing of issues like in the below example:
$ make check
[100%] running cppcheck
[/opt/work/src/main.c:69]: (style) Variable 'i' is allocated memory that is never used.
[/opt/work/src/main.c:73]: (error) Memory leak: i
CMakeFiles/check.dir/build.make:57: recipe for target 'CMakeFiles/check' failed
make[3]: *** [CMakeFiles/check] Error 1
CMakeFiles/Makefile2:141: recipe for target 'CMakeFiles/check.dir/all' failed
make[2]: *** [CMakeFiles/check.dir/all] Error 2
CMakeFiles/Makefile2:148: recipe for target 'CMakeFiles/check.dir/rule' failed
make[1]: *** [CMakeFiles/check.dir/rule] Error 2
Makefile:155: recipe for target 'check' failed
make: *** [check] Error 2
This is a very useful tool used during development.
Code Style Check and Correction
imonitor comes with automatic code style checking and correction. To check if the source code conforms to the project coding style, execute following:
make format-check
If there are no issues, output from the code style checker will be empty. Otherwise, above command will fail like in the below example:
$ make format-check
[100%] Checking clang-format changes
CMakeFiles/format-check.dir/build.make:57: recipe for target 'CMakeFiles/format-check' failed
make[3]: *** [CMakeFiles/format-check] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/format-check.dir/all' failed
make[2]: *** [CMakeFiles/format-check.dir/all] Error 2
CMakeFiles/Makefile2:74: recipe for target 'CMakeFiles/format-check.dir/rule' failed
make[1]: *** [CMakeFiles/format-check.dir/rule] Error 2
Makefile:173: recipe for target 'format-check' failed
make: *** [format-check] Error 2
You can even use the same code style checker to automatically format the source code so that it conforms to the project code style using:
make format
Above command will do the an in-place modification of a project source code. This can be used to automatically format the source code after you have done with your changes.
NOTE: Since make format will do an in-place modification, ensure that you execute this command with the user user. Otherwise, make format will also change owner of files it modifies to root.
Automated Testing
Ensure that you followed Build Instructions before following this chapter, with one difference. You need to generate a debug version of a Makefile using cmake .. -DCMAKE_BUILD_TYPE=Debug instead of cmake .. -DCMAKE_BUILD_TYPE=Release. Or in full:
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Debug
make
Integration Tests
imonitor comes with written automated tests. Tests are written using CMocka and, besides checking of a business logic, are also validating absence of memory leaks using Valgrind. To run those automated tests, execute following in the project $ROOT/build:
sudo make test
If everything goes OK, output would be something like this:
$ sudo make test
Running tests...
Test project /opt/work/build
Start 1: uci_parser
1/6 Test #1: uci_parser ....................... Passed 0.00 sec
Start 2: monitor
2/6 Test #2: monitor .......................... Passed 15.06 sec
Start 3: control
3/6 Test #3: control .......................... Passed 2.01 sec
Start 4: uci_parser_valgrind
4/6 Test #4: uci_parser_valgrind .............. Passed 0.55 sec
Start 5: monitor_valgrind
5/6 Test #5: monitor_valgrind ................. Passed 15.68 sec
Start 6: control_valgrind
6/6 Test #6: control_valgrind ................. Passed 2.61 sec
100% tests passed, 0 tests failed out of 6
Total Test time (real) = 35.93 sec
Coverage Report
Coverage report is also available. Report is based on the integration tests. To generate coverage report, execute following:
sudo make test_coverage
Output from this target is something like this:
# sudo make test_coverage
[ 83%] Built target imonitor-api
[100%] Resetting code coverage counters to zero.
Processing code coverage counters and generating report.
Deleting all .da files in . and subdirectories
Done.
Capturing coverage data from .
Found gcov version: 5.4.0
Scanning . for .gcno files ...
Found 11 graph files in .
Processing CMakeFiles/imonitor-api.dir/src/main.c.gcno
Processing CMakeFiles/imonitor-api.dir/src/monitor.c.gcno
Processing CMakeFiles/imonitor-api.dir/src/uci_parser.c.gcno
Processing CMakeFiles/imonitor-api.dir/src/control.c.gcno
Processing CMakeFiles/imonitor.dir/src/main.c.gcno
Processing CMakeFiles/imonitor.dir/src/monitor.c.gcno
Processing CMakeFiles/imonitor.dir/src/uci_parser.c.gcno
Processing CMakeFiles/imonitor.dir/src/control.c.gcno
Processing test/CMakeFiles/control.dir/control.c.gcno
Processing test/CMakeFiles/monitor.dir/monitor.c.gcno
Processing test/CMakeFiles/uci_parser.dir/uci_parser.c.gcno
geninfo: Note: --initial does not generate branch coverage data
Finished .info-file creation
Test project /opt/work/build
Start 1: uci_parser
1/6 Test #1: uci_parser ....................... Passed 0.00 sec
Start 2: monitor
2/6 Test #2: monitor .......................... Passed 15.07 sec
Start 3: control
3/6 Test #3: control .......................... Passed 2.02 sec
Start 4: uci_parser_valgrind
4/6 Test #4: uci_parser_valgrind .............. Passed 0.56 sec
Start 5: monitor_valgrind
5/6 Test #5: monitor_valgrind ................. Passed 15.78 sec
Start 6: control_valgrind
6/6 Test #6: control_valgrind ................. Passed 2.89 sec
100% tests passed, 0 tests failed out of 6
Total Test time (real) = 36.32 sec
Capturing coverage data from .
Found gcov version: 5.4.0
Scanning . for .gcda files ...
Found 7 data files in .
Processing CMakeFiles/imonitor-api.dir/src/monitor.c.gcda
Processing CMakeFiles/imonitor-api.dir/src/main.c.gcda
Processing CMakeFiles/imonitor-api.dir/src/uci_parser.c.gcda
Processing CMakeFiles/imonitor-api.dir/src/control.c.gcda
Processing test/CMakeFiles/control.dir/control.c.gcda
Processing test/CMakeFiles/monitor.dir/monitor.c.gcda
Processing test/CMakeFiles/uci_parser.dir/uci_parser.c.gcda
Finished .info-file creation
Combining tracefiles.
Reading tracefile test_coverage.base
Reading tracefile test_coverage.info
Writing data to test_coverage.total
Summary coverage rate:
lines......: 92.1% (676 of 734 lines)
functions..: 93.5% (86 of 92 functions)
branches...: 63.9% (131 of 205 branches)
Reading tracefile test_coverage.total
Removing /opt/work/src/main.c
Removing /usr/include/libubox/blob.h
Removing /usr/include/libubox/blobmsg.h
Removing /usr/include/libubox/list.h
Removing /usr/include/libubox/uloop.h
Removing /usr/include/libubox/utils.h
Removing /usr/include/libubus.h
Removing /usr/include/uci.h
Deleted 8 files
Writing data to /opt/work/build/test_coverage.info.cleaned
Summary coverage rate:
lines......: 94.2% (616 of 654 lines)
functions..: 95.7% (66 of 69 functions)
branches...: 64.6% (122 of 189 branches)
Reading data file /opt/work/build/test_coverage.info.cleaned
Found 6 entries.
Found common filename prefix "/opt/work"
Writing .css and .png files.
Generating output.
Processing file src/uci_parser.c
Processing file src/monitor.c
Processing file src/control.c
Processing file test/control.c
Processing file test/uci_parser.c
Processing file test/monitor.c
Writing directory view page.
Overall coverage rate:
lines......: 94.2% (616 of 654 lines)
functions..: 95.7% (66 of 69 functions)
branches...: 64.6% (122 of 189 branches)
Open ./test_coverage/index.html in your browser to view the coverage report.
[100%] Built target test_coverage
Summary can be seen in terminal but if you want the detailed report, open the $ROOT/build/test_coverage/index.html your web browser.
Development Workflow
Use your preferred text editor or IDE to make the necessary implementation changes in the source code. Don't forget to also write a test cases that are proving that your change is working correctly.
If you want to have a full development experience with the code completion, debugging and many more great features which a full-blown IDE can provide, you can use NetBeans IDE. NetBeans IDE supports a remote development feature. The idea is to have a NetBeans IDE running on your host, while everything else is running on the prepared Docker container. To use this feature, connect NetBeans IDE to the running container over SSH. More details on this can be found in C/C++ Remote Development - NetBeans IDE Tutorial.
After you have made your changes, run following commands:
make
make check
sudo make test
If each and every command from above is executed without reporting errors, than you just need to format the source code against the project code style. To do this, run following:
make format
Your changes are now ready for committing into Git.