For questions about the API, have a look at the API documentation page
Go to Splunk and configure in order to be able to receive logs(intelmq events) to a TCP port
Use TCP output bot and configure accordingly to the Splunk configuration that you applied.
If you get an error like this:
intelmq.lib.exceptions.PipelineError: pipeline failed - ConnectionError('Error 13 connecting to unix socket: /var/run/redis/redis.sock. Permission denied.',)
Make sure the intelmq user as sufficient permissions for the socket.
/etc/redis/redis.conf (or wherever your configuration is), check the permissions and set it for example to group-writeable:
And add the user intelmq to the redis-group:
usermod -aG redis intelmq
If you wonder why you are getting errors like this:
intelmq.lib.exceptions.InvalidValue: invalid value '2017-03-06T07:36:29' () for key 'time.source'
IntelMQ requires time zone information for all timestamps. Without a time zone, the time is ambiguous and therefore rejected.
In most cases the bottlenecks are look-up experts. In these cases you can easily use the integrated load balancing features.
When using the AMQP broker, you can make use of Multi-threading. See the Multithreading (Beta) section.
Before Multithreading was available in IntelMQ, and in case you use Redis as broker, the only way to do load balancing involves more work.
Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter
true for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues and not to all of them.
True Multi*processing* is not available in IntelMQ. See also this discussion on a possible enhanced load balancing.
For any bottleneck based on (online) lookups, optimize the lookup itself and if possible use local databases.
It is also possible to use multiple servers to spread the workload. To get the messages from one system to the other you can either directly connect to the other’s pipeline or use a fast exchange mechanism such as the TCP Collector/Output (make sure to secure the network by other means).
If you do not need the raw data, you can safely remove it. For events (after parsers), it keeps the original data, eg. a line of a CSV file. In reports it keeps the actual data to be parsed, so don’t delete the raw field in Reports - between collectors and parsers.
The raw data consumes about 50% - 30% of the messages’ size. The size of course depends on how many additional data you add to it and how much data the report includes. Dropping it, will improve the speed as less data needs to be transferred and processed at each step.
In a bot
You can do this for example by using the Field Reducer Expert. The configuration could be:
Other solutions are the Modify bot and the Sieve bot. The last one is a good choice if you already use it and you only need to add the command:
In the database
In case you store data in the database and you want to keep its size small, you can (periodically) delete the raw data there.
To remove the raw data for a events table of a PostgreSQL database, you can use something like:
UPDATE events SET raw = NULL WHERE "time.source" < '2018-07-01';
If the database is big, make sure only update small parts of the database by using an appropriate
WHERE clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The
id column can also be used instead of the source’s time.
Another way of reducing the
raw-data from the database is described in the EventDB documentation: Separating raw values in PostgreSQL using view and trigger
Rather than starting your bot(s) with
intelmqctl start, try
intelmqctl run [bot]. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.
This section has been moved to the section Orphaned Queues.
Multithreading is not available for some bots and AMQP broker is necessary. Possible reasons why a certain bot or a setup does not support Multithreading include:
Multithreading is only available when using the AMQP broker.
For most collectors, Multithreading is disabled. Otherwise this would lead to duplicated data, as the data retrieval is not atomic.
Some bots use libraries which are not thread safe. Look a the bot’s documentation for more information.
Some bots’ operations are not thread safe. Look a the bot’s documentation for more information.
If you think this mapping is wrong, please report a bug.
If you run our docker image in production, we recommend you to set security headers.
You can do this by creating a new file called
example_config/nginx/security.conf in the cloned
Write the following inside the configuration file, and change the
http(s)://<your-domain> to your domain name.
server_tokens off; # turn off server_token, instead of nginx/13.2 now it will only show nginx add_header X-Frame-Options SAMEORIGIN; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Frame-Options add_header X-Content-Type-Options nosniff; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options add_header X-XSS-Protection "1; mode=block"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection add_header Content-Security-Policy "script-src 'self' 'unsafe-inline' http(s)://<your-domain>; frame-src 'self' http(s)://<your-domain>; object-src 'self' http(s)://<your-domain>"; # https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
After you created the file, edit the
docker-compose.yml and mount it to the
volumes: - ./example_config/nginx/security.conf:/etc/nginx/conf.d/security.conf
IMPORTANT Mount the exact name & not the directory, because otherwise you would overwrite the whole directory and the other files would be gone inside the container.