Frequently asked questions ¶

For questions about the API, have a look at the API documentation page

Send IntelMQ events to Splunk ¶

Go to Splunk and configure in order to be able to receive logs(intelmq events) to a TCP port
Use TCP output bot and configure accordingly to the Splunk configuration that you applied.

Permission denied when using Redis Unix socket ¶

If you get an error like this:

intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 13 connecting to unix socket: /var/run/redis/redis.sock. Permission denied.',)

Make sure the intelmq user as sufficient permissions for the socket.

In /etc/redis/redis.conf (or wherever your configuration is), check the permissions and set it for example to group-writeable:

unixsocketperm 770

And add the user intelmq to the redis-group:

usermod -aG redis intelmq

Why is the time invalid?¶

If you wonder why you are getting errors like this:

intelmq.lib.exceptions.InvalidValue: invalid value '2017-03-06T07:36:29' () for key 'time.source'

IntelMQ requires time zone information for all timestamps. Without a time zone, the time is ambiguous and therefore rejected.

How can I improve the speed?¶

In most cases the bottlenecks are look-up experts. In these cases you can easily use the integrated load balancing features.

Multithreading ¶

When using the AMQP broker, you can make use of Multi-threading. See the Multithreading (Beta) section.

“Classic” load-balancing (Multiprocessing)¶

Before Multithreading was available in IntelMQ, and in case you use Redis as broker, the only way to do load balancing involves more work. Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter load_balance to true for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues and not to all of them.

True Multi*processing* is not available in IntelMQ. See also this discussion on a possible enhanced load balancing.

Other options ¶

For any bottleneck based on (online) lookups, optimize the lookup itself and if possible use local databases.

It is also possible to use multiple servers to spread the workload. To get the messages from one system to the other you can either directly connect to the other’s pipeline or use a fast exchange mechanism such as the TCP Collector/Output (make sure to secure the network by other means).

Removing raw data for higher performance and less space usage ¶

If you do not need the raw data, you can safely remove it. For events (after parsers), it keeps the original data, eg. a line of a CSV file. In reports it keeps the actual data to be parsed, so don’t delete the raw field in Reports - between collectors and parsers.

The raw data consumes about 50% - 30% of the messages’ size. The size of course depends on how many additional data you add to it and how much data the report includes. Dropping it, will improve the speed as less data needs to be transferred and processed at each step.

In a bot

You can do this for example by using the Field Reducer Expert. The configuration could be:

type: blacklist

keys: raw

Other solutions are the Modify bot and the Sieve bot. The last one is a good choice if you already use it and you only need to add the command:

remove raw

In the database

In case you store data in the database and you want to keep its size small, you can (periodically) delete the raw data there.

To remove the raw data for a events table of a PostgreSQL database, you can use something like:

UPDATE events SET raw = NULL WHERE "time.source" < '2018-07-01';

If the database is big, make sure only update small parts of the database by using an appropriate WHERE clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The id column can also be used instead of the source’s time.

Another way of reducing the raw-data from the database is described in the EventDB documentation: Separating raw values in PostgreSQL using view and trigger

My bot(s) died on startup with no errors logged ¶

Rather than starting your bot(s) with intelmqctl start, try intelmqctl run [bot]. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.

Orphaned Queues ¶

This section has been moved to the section Orphaned Queues.

Multithreading is not available for this bot ¶

Multithreading is not available for some bots and AMQP broker is necessary. Possible reasons why a certain bot or a setup does not support Multithreading include:

Multithreading is only available when using the AMQP broker.

For most collectors, Multithreading is disabled. Otherwise this would lead to duplicated data, as the data retrieval is not atomic.

Some bots use libraries which are not thread safe. Look a the bot’s documentation for more information.

Some bots’ operations are not thread safe. Look a the bot’s documentation for more information.

If you think this mapping is wrong, please report a bug.

Docker: Security Headers ¶

If you run our docker image in production, we recommend you to set security headers. You can do this by creating a new file called example_config/nginx/security.conf in the cloned intelmq-docker repository.

Write the following inside the configuration file, and change the http(s)://<your-domain> to your domain name.

server_tokens off; # turn off server_token, instead of nginx/13.2 now it will only show nginx
add_header X-Frame-Options SAMEORIGIN; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options
add_header X-Content-Type-Options nosniff; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options
add_header X-XSS-Protection "1; mode=block"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection
add_header Content-Security-Policy "script-src 'self' 'unsafe-inline' http(s)://<your-domain>; frame-src 'self' http(s)://<your-domain>; object-src 'self' http(s)://<your-domain>"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP

After you created the file, edit the docker-compose.yml and mount it to the nginx with

volumes:
- ./example_config/nginx/security.conf:/etc/nginx/conf.d/security.conf

IMPORTANT Mount the exact name & not the directory, because otherwise you would overwrite the whole directory and the other files would be gone inside the container.